Glossary
k-Shingles
Fundamentals
Adversarial Machine LearningAI AgentsAI and EducationAI and MedicineAI AssistantsAI HardwareAI Voice TransferApproximate Dynamic ProgrammingBackpropagationBayesian Machine LearningCurse of DimensionalityData LabelingDeep LearningDeep Reinforcement LearningDimensionality ReductionF1 Score in Machine LearningFeedforward Neural NetworkFine Tuning in Deep LearningGated Recurrent UnitGenerative AILarge Language Model (LLM)Machine LearningMultimodal LearningNatural Language Generation (NLG)Natural Language Processing (NLP)Natural Language Understanding (NLU)Precision and RecallTransformersZero-shot Classification Models
Models
AlphaGo ZeroBERTChatGPTChess botsDall-EDiffusion ModelsDistilBERTGoogle's BardHidden Markov Models (HMMs)Inference EngineLarge Language Model (LLM)Llama 2LLM CollectionMidjourney (Image Generation)MistralOpenAI WhisperPerceptronProbabilistic Models in Machine LearningRoBERTaRule-Based AISpeech-to-text modelsText-to-Speech ModelsXLNet
Techniques
Activation FunctionsBatch Gradient DescentBeam Search AlgorithmBenchmarkingClassificationCURE AlgorithmExpectation MaximizationFlajolet-Martin AlgorithmGaussian ProcessesGenerative Adversarial Networks (GANs)Gradient Boosting Machines (GBMs)Gradient ClippingGradient ScalingGroundingHyperparametersk-ShinglesLatent Dirichlet Allocation (LDA)Mixture of ExpertsMultimodal AIOnline Gradient DescentOverfitting and UnderfittingPrompt ChainingPrompt EngineeringRetrieval-Augmented Generation (RAG)RLHFSentiment AnalysisSequence ModelingSemantic KernelTokenizationWinnow AlgorithmWord Embeddings
Last updated on January 25, 20249 min read
k-Shingles

Have you ever pondered how search engines identify similar content, or how plagiarism detectors work their magic? At the heart of these technologies lies a fascinating concept: Jaccard Similarity and k-Shingles.

Have you ever pondered how search engines identify similar content, or how plagiarism detectors work their magic? At the heart of these technologies lies a fascinating concept: Jaccard Similarity and k-Shingles. Imagine you have two sets of data, and you wish to know how closely they resemble each other. This is where Jaccard Similarity, a statistical measure, comes into play, offering a mathematical approach to understanding similarity and diversity. Now, couple that with k-Shingles, a technique in text mining that transforms chunks of text into comparable sets. The blend of these two methodologies provides a powerful tool for various applications, from SEO to bioinformatics.

Section 1: What is Jaccard Similarity and k-Shingles?

The Jaccard similarity of sets S and T is |S ∩ T| / |S ∪ T|, that is, the ratio of the size of the intersection of S and T to the size of their union. You can denote the Jaccard similarity of S and T by sim(S, T).