FareedKhan/flax-sentence-embeddings_all_datasets_v4_MiniLM-L6_FareedKhan_prime_synthetic_data_2k_10_32

SentenceTransformer based on flax-sentence-embeddings/all_datasets_v4_MiniLM-L6

This is a sentence-transformers model finetuned from flax-sentence-embeddings/all_datasets_v4_MiniLM-L6 on the json dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: flax-sentence-embeddings/all_datasets_v4_MiniLM-L6
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 384 tokens
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("FareedKhan/flax-sentence-embeddings_all_datasets_v4_MiniLM-L6_FareedKhan_prime_synthetic_data_2k_10_32")
# Run inference
sentences = [
    '\nAtypical hemolytic uremic syndrome (aHUS) with H factor anomaly is a disease characterized by an atypical form of hemolytic uremic syndrome, a severe thrombotic microangiopathy that leads to kidney failure, anemia, and thrombocytopenia. This specific subtype of aHUS is notable for its association with an anomaly in the H factor, potentially involving complement system dysregulation. As such, it falls under the broader category of hemolytic uremic syndrome, a condition marked by differential diagnosis complexity and distinct etiologies. Patients with aHUS often require a nuanced approach to diagnosis and management, emphasizing awareness of its distinct characteristics in comparison with other forms of hemolytic uremic syndrome, ensuring comprehensive and accurate differential diagnosis which might include conditions like thrombotic thrombocytopenic purpura (TTP) or disseminated intravascular coagulation (DIC). The identification and management of aHUS with H factor anomaly necessitates multidisciplinary collaboration and up-to-date knowledge alongside genetic and clinical features specific to this condition.',
    'Could you list the diseases related to or subtypes of type 1 atypical hemolytic uremic syndrome for differential diagnosis purposes?',
    'Which diseases are associated with anomalies in the CD4 gene or protein, alongside genetic mutations that impact muscle protein synthesis?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_384
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.3861
cosine_accuracy@3	0.4604
cosine_accuracy@5	0.4901
cosine_accuracy@10	0.5149
cosine_precision@1	0.3861
cosine_precision@3	0.1535
cosine_precision@5	0.098
cosine_precision@10	0.0515
cosine_recall@1	0.3861
cosine_recall@3	0.4604
cosine_recall@5	0.4901
cosine_recall@10	0.5149
cosine_ndcg@10	0.4514
cosine_mrr@10	0.4312
cosine_map@100	0.4383

Training Details

Training Dataset

json

• Dataset: json
• Size: 1,814 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 3 tokens mean: 118.96 tokens max: 128 tokens	min: 14 tokens mean: 34.93 tokens max: 116 tokens

• Samples:

  positive	anchor
  Epilepsy is a neurological disorder characterized by recurrent seizures, which can be sudden, abnormal electrical events in the brain. Seizures can affect different parts of the brain and range from mild to severe. Symptoms can include muscle stiffness, twitching, loss of consciousness, and cognitive disruptions. Seizures can be divided into focal (partial) seizures and generalized seizures. ### Causes: 1. Brain Injury or Infection: These can lead to scar tissue and abnormal electrical activity. 2. Developmental Abnormalities: Genetic or developmental issues can cause seizures. 3. Brain Tumors or Bleeds: These can result in seizures. 4. Inherited Disorders: Some genetic conditions lead to epilepsy. ### Complications: - Status Epilepticus: Continuous seizure activity lasting more than five minutes. - Sudden Unexpected Death in Epilepsy (SUDEP): Unexplained death during an untreated condition, especially if seizures aren't controlled. - Emotional Health Issues: Increased risk for depression, anxiety, and	Search for medical conditions not treatable by any known medications that present with hoarseness as a symptom.
  Diphyllobothriasis, also known as bothriocephalosis, is a parasitosis caused by the intestinal infection with the larval stage of the tapeworm Diphyllobothrium. This condition is characterized by a broad array of symptoms, including frequent stomach discomfort, nausea, appetite loss, fatigue, and weakness. These symptoms are medically attributed to anemia, which stems from vitamin B12 deficiency—a common complication linked to this parasitosis. The anemia caused by diphyllobothriasis can also resemble Biermer's anemia, distinguished by abnormally large red blood cells. Individuals with a family history of ceestode infections, such as diphyllobothriasis, and those who exhibit symptoms such as those described, may be more susceptible to this condition. The disease, which is cosmopolitan in nature, has been reported in Europe, primarily in areas like the Italian, Swiss, and French Alps, though its prevalence across the continent remains unknown. Treatment for diphyllobothriasis typically involves the use of standard medications such as niclosamide or praziquantel, which are effective in clearing the parasite.	What could be the condition causing frequent stomach discomfort, nausea, appetite loss, fatigue, and weakness in me, possibly linked to a family history of Cestode infection and associated with vitamin B12 deficiency and abnormal red blood cells resembling Biermer's anemia symptoms?
  The provided list appears to be a collection of gene names. Genes are segments of DNA that code for proteins and play a crucial role in various biological functions, influencing traits, growth, and processes within an organism. They are fundamental units of heredity. The presence of these gene names suggests that the document is most likely related to genetic research, medical studies, or bioinformatics. This could involve analyses of genetic sequences, expression patterns, or functional assays related to the specific genes mentioned, possibly with the aim of understanding genetic disorders, development, or disease mechanisms.	Which cellular structures engage in interactions with genes or proteins that are affected by the administration of Mevastatin?

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          384
      ],
      "matryoshka_weights": [
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• learning_rate: 1e-05
• num_train_epochs: 10
• warmup_ratio: 0.1
• bf16: True
• tf32: False
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_384_cosine_map@100
0	0	-	0.3748
0.1754	10	1.5606	-
0.3509	20	1.5914	-
0.5263	30	1.6623	-
0.7018	40	1.7258	-
0.8772	50	1.6031	-
1.0	57	-	0.4241
1.0526	60	1.4494	-
1.2281	70	1.4091	-
1.4035	80	1.3177	-
1.5789	90	1.3299	-
1.7544	100	1.459	-
1.9298	110	1.3534	-
2.0	114	-	0.4214
2.1053	120	1.3023	-
2.2807	130	1.2222	-
2.4561	140	1.2191	-
2.6316	150	1.0443	-
2.8070	160	1.1894	-
2.9825	170	1.0955	-
3.0	171	-	0.4156
3.1579	180	1.1698	-
3.3333	190	0.9699	-
3.5088	200	1.0524	-
3.6842	210	0.9902	-
3.8596	220	1.0943	-
4.0	228	-	0.4221
4.0351	230	0.9793	-
4.2105	240	0.9786	-
4.3860	250	1.0352	-
4.5614	260	0.9809	-
4.7368	270	0.8568	-
4.9123	280	0.9372	-
5.0	285	-	0.4264
5.0877	290	0.8529	-
5.2632	300	0.9472	-
5.4386	310	0.8436	-
5.6140	320	0.8166	-
5.7895	330	0.8731	-
5.9649	340	0.9489	-
6.0	342	-	0.4274
6.1404	350	0.9991	-
6.3158	360	0.7533	-
6.4912	370	0.9122	-
6.6667	380	0.8404	-
6.8421	390	0.7928	-
7.0	399	-	0.4302
7.0175	400	0.8332	-
7.1930	410	0.7534	-
7.3684	420	0.8424	-
7.5439	430	0.8465	-
7.7193	440	0.8461	-
7.8947	450	0.7203	-
8.0	456	-	0.4344
8.0702	460	0.8144	-
8.2456	470	0.7895	-
8.4211	480	0.7665	-
8.5965	490	0.883	-
8.7719	500	0.6908	-
8.9474	510	0.8481	-
9.0	513	-	0.4365
9.1228	520	0.7521	-
9.2982	530	0.6971	-
9.4737	540	0.7081	-
9.6491	550	0.8272	-
9.8246	560	0.7922	-
10.0	570	0.7998	0.4383

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.10
• Sentence Transformers: 3.1.1
• Transformers: 4.45.1
• PyTorch: 2.2.1+cu121
• Accelerate: 0.34.2
• Datasets: 3.0.1
• Tokenizers: 0.20.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}