Module documentation

elfen.configs module

Submodules

elfen.custom module

Generic functions for custom lexicon feature extraction. These functions allow the user to extract features from a custom lexicon. The lexicon should be a DataFrame with with the tokens or lemmas in one column and the features (e.g. concreteness; only if thresholds are to be applied) in another column.

Functions:

get_n_custom: Get the number of occurences of words in a custom lexicon: in the text.
get_occurs_custom: Binary feature indicating if any word from a custom: lexicon occurs in the text.
get_n_custom_low: Get the number of occurences of words with a low: given feature (e.g. concreteness) from a custom lexicon in the text.
get_n_custom_high: Get the number of occurences of words with a high: given feature (e.g. concreteness) from a custom lexicon in the text.

elfen.custom.get_avg_custom(data: DataFrame, lexicon: DataFrame, feature_name: str = 'avg_custom', word_column: str = 'word', feature_column: str = 'feature') → DataFrame

Get the average feature value of words in a custom lexicon in the text.

Parameters:

data (pl.DataFrame) – The data to process. Should contain the preprocessed column ‘nlp’.
lexicon (pl.DataFrame) – The lexicon to use.
feature_name (str) – The name of the feature to create.
word_column (str) – The column the words occur in in the lexicon.
feature_column (str) – The column the feature occurs in in the lexicon.
measurement_level (str) – The measurement level of the lexicon; either ‘tokens’ or ‘lemmas’.

Returns:

The data with the new feature.

Return type:

data (pl.DataFrame)

elfen.custom.get_n_custom(data: DataFrame, lexicon: DataFrame, feature_name: str = 'n_custom', word_column: str = 'word', measurement_level: str = 'tokens') → DataFrame

Get the number of occurences of words in a custom lexicon in the text.

Parameters:

data (pl.DataFrame) – The data to process. Should contain the preprocessed column ‘nlp’.
lexicon (pl.DataFrame) – The lexicon to use.
feature_name (str) – The name of the feature to create.
word_column (str) – The column the words occur in in the lexicon.
measurement_level (str) – The measurement level of the lexicon; either ‘tokens’ or ‘lemmas’.

Returns:

The data with the new feature.

Return type:

data (pl.DataFrame)

elfen.custom.get_n_custom_high(data: DataFrame, lexicon: DataFrame, threshold: int, feature_name: str = 'n_custom_high', word_column: str = 'word', feature_column: str = 'feature') → DataFrame

Get the number of occurences of words in a custom lexicon in the text with a given feature (e.g. concreteness) above a threshold.

Parameters:

data (pl.DataFrame) – The data to process. Should contain the preprocessed column ‘nlp’.
lexicon (pl.DataFrame) – The lexicon to use.
threshold (int) – The threshold to use.
feature_name (str) – The name of the feature to create.
word_column (str) – The column the words occur in in the lexicon.
feature_column (str) – The column the feature occurs in in the lexicon.
measurement_level (str) – The measurement level of the lexicon; either ‘tokens’ or ‘lemmas’.

Returns:

The data with the new feature.

Return type:

data (pl.DataFrame)

elfen.custom.get_n_custom_low(data: DataFrame, lexicon: DataFrame, threshold: int, feature_name: str, word_column: str = 'word', feature_column: str = 'feature') → DataFrame

Get the number of occurences of words in a custom lexicon in the text with a given feature (e.g. concreteness) below a threshold.

Parameters:

data (pl.DataFrame) – The data to process. Should contain the preprocessed column ‘nlp’.
lexicon (pl.DataFrame) – The lexicon to use.
feature_name (str) – The name of the feature to create.
threshold (int) – The threshold to use.
word_column (str) – The column the words occur in in the lexicon.
feature_column (str) – The column the feature occurs in in the lexicon.

Returns:

The data with the new feature in the column specified by feature_name.

Return type:

data (pl.DataFrame)

elfen.custom.get_occurs_custom(data: DataFrame, lexicon: DataFrame, feature_name: str = 'occurs_custom', word_column: str = 'word', measurement_level: str = 'tokens') → DataFrame

Binary feature indicating if any word from a custom lexicon occurs in the text.

Parameters:

data (pl.DataFrame) – The data to process. Should contain the preprocessed column ‘nlp’.
lexicon (pl.DataFrame) – The lexicon to use.
feature_name (str) – The name of the feature to create.
word_column (str) – The column the words occur in in the lexicon.

Returns:

The data with the new feature.

Return type:

data (pl.DataFrame)

elfen.dependency module

This module contains functions to extract dependency features from the input text. Dependency features are based on the dependency tree of the text data. The dependency tree is a directed graph that represents the syntactic structure of the text. Each node in the graph represents a token in the text, and each edge represents a dependency relation between two tokens. Dependency features can be used to capture the syntactic complexity of the text data.

The following dependency features are implemented in this module:

Tree width:
The maximum number of nodes in the dependency tree of a token.
Tree depth:
The maximum distance of a token from the root of the dependency tree.
Tree branching factor:
The average number of children of a token.
Ramification factor:
The mean number of children per level in the dependency tree.
Number of noun chunks:
The number of noun chunks in the text.
Frequency per dependency type:
The frequency of each dependency type in the text.

elfen.dependency.get_n_noun_chunks(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Extracts the number of noun chunks in the input text.

Parameters:

data (pl.DataFrame) – A polars dataframe containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

The input data with the number of noun chunks stored in a new column named ‘n_noun_chunks’.

Return type:

data (pl.DataFrame)

elfen.dependency.get_n_per_dependency_type(data: DataFrame, backbone: str = 'spacy', language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Extracts the frequency per dependency type in the input text.

Parameters:

data (pl.DataFrame) – A polars dataframe containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
language (str) – The language of the text data. E.g. ‘en’, ‘de’.

Returns:

The input data with the frequency per dependency type stored in new columns named ‘n_dependency_{dep}’.

Return type:

data (pl.DataFrame)

elfen.dependency.get_ramification_factor(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Extracts the dependency tree ramification factor of the input text. The ramification factor is the mean number of children per level.

Parameters:

data (pl.DataFrame) – A polars dataframe containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

The input data with the dependency tree ramification factor stored in a new column named ‘ramification_factor’.

Return type:

data (pl.DataFrame)

elfen.dependency.get_tree_branching(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Extracts the dependency tree branching factor of the input text. The branching factor is the average number of children of a token.

Parameters:

data (pl.DataFrame) – A polars dataframe containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

The input data with the dependency tree branching factor stored in a new column named ‘tree_branching’.

Return type:

data (pl.DataFrame)

elfen.dependency.get_tree_depth(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Extracts the dependency tree depth of the input text. Tree depth is the maximum distance of a token from the root of the dependency tree. If the input text contains multiple sentences, the average tree depth is calculated.

Parameters:

data (pl.DataFrame) – A polars dataframe containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

The input data with the dependency tree depth stored in a new column named ‘tree_depth’.

Return type:

data (pl.DataFrame)

elfen.dependency.get_tree_width(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Extracts the dependency tree width of the input text. Tree width is the maximum number of nodes in the dependency tree of a token.

Parameters:

data (pl.DataFrame) – A polars dataframe containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

The input data with the dependency tree width stored in a new column named ‘tree_width’.

Return type:

data (pl.DataFrame)

elfen.emotion module

This module contains functions to calculate the VAD dimensions, Plutchik emotions, the sentiment and emotion intensity of text data.

If you are using this functionality, please cite the original authors of the resources.

The lexicons used in this module are:

NRC VAD Lexicon:
Mohammad, S. M., & Turney, P. D. (2013). Crowdsourcing a Word-Emotion Association Lexicon. Computational Intelligence, 29(3), 436-465.
NRC Emotion Intensity Lexicon:
Mohammad, S. M. (2018). Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 174-184).
NRC Sentiment Lexicon:
Mohammad, S.M. (2018). Word Affect Intensities. In Proceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018), May 2018, Miyazaki, Japan.

The VAD lexicon contains three dimensions: Valence, Arousal, and Dominance.

The emotion intensity lexicon contains the intensity of eight Plutchik emotions: Anger, Anticipation, Disgust, Fear, Joy, Sadness, Surprise, and Trust.

The sentiment lexicon contains the sentiment of words: Positive and Negative.

The following functions are implemented in this module:

VAD Dimensions:
- load_vad_lexicon:
  Loads the VAD lexicon as a polars DataFrame.
- get_avg_valence:
  Calculates the average valence of the text.
- get_avg_arousal:
  Calculates the average arousal of the text.
- get_avg_dominance:
  Calculates the average dominance of the text.
- get_n_low_valence
  Calculates the number of words with valence lower than the threshold.
- get_n_high_valence:
  Calculates the number of words with valence higher than the threshold.
- get_n_low_arousal:
  Calculates the number of words with arousal lower than the threshold.
- get_n_high_arousal:
  Calculates the number of words with arousal higher than the threshold.
- get_n_low_dominance:
  Calculates the number of words with dominance lower than the threshold.
- get_n_high_dominance:
  Calculates the number of words with dominance higher than the threshold.
- get_max_valence:
  Calculates the maximum valence of the text.
- get_min_valence:
  Calculates the minimum valence of the text.
- get_sd_valence:
  Calculates the standard deviation of valence of the text.
- get_max_arousal:
  Calculates the maximum arousal of the text.
- get_min_arousal:
  Calculates the minimum arousal of the text.
- get_sd_arousal:
  Calculates the standard deviation of arousal of the text.
- get_max_dominance:
  Calculates the maximum dominance of the text.
- get_min_dominance:
  Calculates the minimum dominance of the text.
- get_sd_dominance:
  Calculates the standard deviation of dominance of the text.
Emotion Intensity:
- load_intensity_lexicon:
  Loads the emotion intensity lexicon as a polars DataFrame.
- get_avg_emotion_intensity:
  Calculates the average emotion intensity of the text.
- get_n_low_intensity:
  Calculates the number of words with emotion intensity lower than the threshold.
- get_n_high_intensity:
  Calculates the number of words with emotion intensity higher than the threshold.
- get_max_emotion_intensity:
  Calculates the maximum emotion intensity of the text.
- get_min_emotion_intensity:
  Calculates the minimum emotion intensity of the text.
- get_sd_emotion_intensity:
  Calculates the standard deviation of emotion intensity of the text.
Sentiment Analysis:
- load_sentiment_nrc_lexicon:
  Loads the sentiment NRC lexicon as a polars DataFrame.
- get_n_positive_sentiment:
  Calculates the number of words with positive sentiment.
- get_n_negative_sentiment:
  Calculates the number of words with negative sentiment.

elfen.emotion.load_intensity_lexicon(path: str = INTENSITY_LEXICON_PATH, schema: dict = INTENSITY_LEXICON_SCHEMA) → DataFrame

Loads the intensity lexicon as a polars DataFrame.

Parameters:

path (str) – The path to the intensity lexicon.
schema (dict) – The schema for the intensity lexicon.
has_header (bool) – Whether the lexicon has a header.

Returns:

The intensity lexicon as a polars DataFrame.

Return type:

intensity_lexicon (pl.DataFrame)

elfen.emotion.load_sentiment_nrc_lexicon(path: str = SENTIMENT_NRC_LEXICON_PATH, schema: dict = SENTIMENT_NRC_LEXICON_SCHEMA) → DataFrame

Loads the sentiment NRC lexicon as a polars DataFrame.

Parameters:

path (str) – The path to the sentiment NRC lexicon.
schema (dict) – The schema for the sentiment NRC lexicon.
has_header (bool) – Whether the lexicon has a header.
separator (str) – The separator used in the lexicon.

Returns:

The sentiment NRC lexicon as a polars DataFrame.

Return type:

sentiment_nrc (pl.DataFrame)

elfen.emotion.load_vad_lexicon(path: str = VAD_LEXICON_PATH, schema: dict = VAD_LEXICON_SCHEMA) → DataFrame

Loads the VAD lexicon as a polars DataFrame.

Parameters:

path (str) – The path to the VAD lexicon.
schema (dict) – The schema for the VAD lexicon.
has_header (bool) – Whether the lexicon has a header.
separator (str) – The separator used in the lexicon.

Returns:

The VAD lexicon as a polars DataFrame.

Return type:

vad_lexicon (pl.DataFrame)

elfen.emotion.filter_intensity_lexicon(lexicon: DataFrame, words: list, emotion: str, word_column: str = 'word', **kwargs: dict[str, str]) → DataFrame

Filters the intensity lexicon for the given words and emotions.

Parameters:

lexicon (pl.DataFrame) – The emotion intensity lexicon.
words (list) – The list of words to filter.
emotion (str) – The emotion to filter for.

Returns:

The filtered emotion intensity lexicon.

Return type:

filtered_intensity_lexicon (pl.DataFrame)

elfen.emotion.filter_sentiment_lexicon(lexicon: DataFrame, words: list, sentiment: str, word_column: str = 'word', **kwargs: dict[str, str]) → DataFrame

Filters the sentiment NRC lexicon for the given words and emotions.

Parameters:

lexicon (pl.DataFrame) – The sentiment lexicon.
words (list) – The list of words to filter.
sentiment (str) – The sentiment to filter for.

Returns:

The filtered sentiment NRC lexicon.

Return type:

filtered_sentiment_nrc (pl.DataFrame)

elfen.emotion.get_avg_arousal(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the average arousal of the text. Only takes into account words in the text that are present in the VAD lexicon.

The arousal of the text is calculated as the mean of the arousal values of the words in the text.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.

Returns:

The input data with the average arousal columns, named “avg_arousal”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_avg_dominance(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the average dominance of the text. Only takes into account words in the text that are present in the VAD lexicon.

The dominance of the text is calculated as the mean of the dominance values of the words in the text.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.
language (str) – The language of the text and lexicon. Defaults to “en”.

Returns:

The input data with the average dominance columns, named “avg_dominance”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_avg_emotion_intensity(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', emotions: list = ['anger', 'anticipation', 'disgust', 'fear', 'joy', 'sadness', 'surprise', 'trust'], language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the average emotion intensity of the text. Only takes into account words in the text that are present in the emotion intensity lexicon.

The average emotion intensity is calculated as the mean of the emotion intensity values of the words in the text.

NaN/Null values are filled with 0 as the emotion intensity is in the range [0,1] and 0 is the neutral value in the NRC emotion intensity lexicon.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The emotion intensity lexicon.
backbone (str) – The NLP backbone to use.
emotions (list) – The list of emotions to consider. Defaults to the Plutchik emotions.

Returns:

The input data with the average emotion intensity columns

Return type:

data (pl.DataFrame)

elfen.emotion.get_avg_valence(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the average valence of the text. Only takes into account words in the text that are present in the VAD lexicon.

The valence of the text is calculated as the mean of the valence values of the words in the text.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.

Returns:

The input data with the average valence columns, named “avg_valence”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_max_arousal(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the maximum arousal of the text. Only takes into account words in the text that are present in the VAD lexicon. The arousal of the text is calculated as the maximum of the arousal values of the words in the text.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.
language (str) – The language of the text and lexicon. Defaults to “en”.

Returns:

The input data with the maximum arousal columns, named “max_arousal”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_max_dominance(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the maximum dominance of the text. Only takes into account words in the text that are present in the VAD lexicon. The dominance of the text is calculated as the maximum of the dominance values of the words in the text.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.
language (str) – The language of the text and lexicon. Defaults to “en”.

Returns:

The input data with the maximum dominance columns, named “max_dominance”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_max_emotion_intensity(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', emotions: list = ['anger', 'anticipation', 'disgust', 'fear', 'joy', 'sadness', 'surprise', 'trust'], language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the maximum emotion intensity of the text. Only takes into account words in the text that are present in the emotion intensity lexicon. The emotion intensity of the text is calculated as the maximum of the emotion intensity values of the words in the text.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The emotion intensity lexicon.
backbone (str) – The NLP backbone to use.
emotions (list) – The list of emotions to consider.

Returns:

The input data with the maximum emotion intensity columns for each emotion. The column names are in the format “max_intensity_{emotion}”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_max_valence(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the maximum valence of the text. Only takes into account words in the text that are present in the VAD lexicon.

The valence of the text is calculated as the maximum of the valence values of the words in the text.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.
language (str) – The language of the text and lexicon. Defaults to “en”.

Returns:

The input data with the maximum valence columns, named “max_valence”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_min_arousal(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the minimum arousal of the text. Only takes into account words in the text that are present in the VAD lexicon. The arousal of the text is calculated as the minimum of the arousal values of the words in the text.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.
language (str) – The language of the text and lexicon. Defaults to “en”.

Returns:

The input data with the minimum arousal columns, named “min_arousal”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_min_dominance(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the minimum dominance of the text. Only takes into account words in the text that are present in the VAD lexicon. The dominance of the text is calculated as the minimum of the dominance values of the words in the text.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.
language (str) – The language of the text and lexicon. Defaults to “en”.

Returns:

The input data with the minimum dominance columns, named “min_dominance”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_min_emotion_intensity(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', emotions: list = ['anger', 'anticipation', 'disgust', 'fear', 'joy', 'sadness', 'surprise', 'trust'], language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the minimum emotion intensity of the text. Only takes into account words in the text that are present in the emotion intensity lexicon. The emotion intensity of the text is calculated as the minimum of the emotion intensity values of the words in the text.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The emotion intensity lexicon.
backbone (str) – The NLP backbone to use.
emotions (list) – The list of emotions to consider.

Returns:

The input data with the minimum emotion intensity columns for each emotion. The column names are in the format “min_intensity_{emotion}”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_min_valence(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the minimum valence of the text. Only takes into account words in the text that are present in the VAD lexicon. The valence of the text is calculated as the minimum of the valence values of the words in the text.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.
language (str)
lexicon. (The language of the text and) – Defaults to “en”.

Returns:

The input data with the minimum valence columns, named “min_valence”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_n_high_arousal(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', threshold: float = 0.66, nan_value: float = 0.0, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the number of words with arousal higher than the threshold.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.
threshold (float) – The threshold for high arousal. Defaults to 0.66.
nan_value (float) – The value to use for NaNs. Defaults to 0.0.

Returns:

The input data with the high arousal count column. The column name is “n_high_arousal”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_n_high_dominance(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', threshold: float = 0.66, nan_value: float = 0.0, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the number of words with dominance higher than the threshold.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.
threshold (float) – The threshold for high dominance. Defaults to 0.66.
nan_value (float) – The value to use for NaNs. Defaults to 0.0.

Returns:

The input data with the high dominance count column. The column name is “n_high_dominance”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_n_high_intensity(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', emotions: list = ['anger', 'anticipation', 'disgust', 'fear', 'joy', 'sadness', 'surprise', 'trust'], threshold: float = 0.66, nan_value: float = 0.0, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the number of words with emotion intensity higher than the threshold.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The emotion intensity lexicon.
backbone (str) – The NLP backbone to use.
emotions (list) – The list of emotions to consider.
threshold (float) – The threshold for high intensity. Defaults to 0.66.
nan_value (float) – The value to use for NaNs. Defaults to 0.0.

Returns:

The input data with the high intensity count column for each emotion. The column names are in the format “n_high_intensity_{emotion}”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_n_high_valence(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', threshold: float = 0.66, nan_value: float = 0.0, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the number of words with valence higher than the threshold.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.
threshold (float) – The threshold for high valence. Defaults to 0.66.
nan_value (float) – The value to use for NaNs. Defaults to 0.0.

Returns:

The input data with the high valence count column, named “n_high_valence”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_n_low_arousal(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', threshold: float = 0.33, nan_value: float = 0.0, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the number of words with arousal lower than the threshold.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.
threshold (float) – The threshold for low arousal. Defaults to 0.33.
nan_value (float) – The value to use for NaNs. Defaults to 0.0.

Returns:

The input data with the low arousal count column, named “n_low_arousal”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_n_low_dominance(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', threshold: float = 0.33, nan_value: float = 0.0, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the number of words with dominance lower than the threshold.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.
threshold (float) – The threshold for low dominance. Defaults to 0.33.
nan_value (float) – The value to use for NaNs. Defaults to 0.0.

Returns:

The input data with the low dominance count column. The column name is “n_low_dominance”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_n_low_intensity(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', emotions: list = ['anger', 'anticipation', 'disgust', 'fear', 'joy', 'sadness', 'surprise', 'trust'], threshold: float = 0.33, nan_value: float = 0.0, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the number of words with emotion intensity lower than the threshold.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The emotion intensity lexicon.
backbone (str) – The NLP backbone to use.
emotions (list) – The list of emotions to consider.
threshold (float) – The threshold for low intensity. Defaults to 0.33.
nan_value (float) – The value to use for NaNs. Defaults to 0.0.

Returns:

The input data with the low intensity count column for each emotion. The column names are in the format “n_low_intensity_{emotion}”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_n_low_valence(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', threshold: float = 0.33, nan_value: float = 0.0, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the number of words with valence lower than the threshold.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.
threshold (float) – The threshold for low valence. Defaults to 0.33.
nan_value (float) – The value to use for NaNs. Defaults to 0.0.

Returns:

The input data with the low valence count column, named “n_low_valence”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_n_negative_sentiment(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', nan_value: float = 0.0, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the number of words with negative sentiment.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The sentiment lexicon.
backbone (str) – The NLP backbone to use.
nan_value (float) – The value to use for NaNs. Defaults to 0.0.

Returns:

The input data with the negative sentiment count column. The column name is “n_negative_sentiment”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_n_positive_sentiment(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', nan_value: float = 0.0, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the number of words with positive sentiment.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The sentiment lexicon.
backbone (str) – The NLP backbone to use.
nan_value (float) – The value to use for NaNs. Defaults to 0.0.

Returns:

The input data with the positive sentiment count column. The column name is “n_positive_sentiment”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_sd_arousal(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the standard deviation of arousal of the text. Only takes into account words in the text that are present in the VAD lexicon. The arousal of the text is calculated as the standard deviation of the arousal values of the words in the text.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.
language (str) – The language of the text and lexicon. Defaults to “en”.

Returns:

The input data with the standard deviation of arousal columns, named “sd_arousal”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_sd_dominance(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the standard deviation of dominance of the text. Only takes into account words in the text that are present in the VAD lexicon. The dominance of the text is calculated as the standard deviation of the dominance values of the words in the text.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.
language (str) – The language of the text and lexicon. Defaults to “en”.

Returns:

The input data with the standard deviation of dominance columns, named “sd_dominance”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_sd_emotion_intensity(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', emotions: list = ['anger', 'anticipation', 'disgust', 'fear', 'joy', 'sadness', 'surprise', 'trust'], language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the standard deviation of emotion intensity of the text. Only takes into account words in the text that are present in the emotion intensity lexicon. The emotion intensity of the text is calculated as the standard deviation of the emotion intensity values of the words in the text.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The emotion intensity lexicon.
backbone (str) – The NLP backbone to use.
emotions (list) – The list of emotions to consider.

Returns:

The input data with the standard deviation of emotion intensity columns for each emotion. The column names are in the format “sd_intensity_{emotion}”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_sd_valence(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the standard deviation of valence of the text. Only takes into account words in the text that are present in the VAD lexicon. The valence of the text is calculated as the standard deviation of the valence values of the words in the text.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The VAD lexicon.
backbone (str) – The NLP backbone to use.
language (str) – The language of the text and lexicon. Defaults to “en”.

Returns:

The input data with the standard deviation of valence columns, named “sd_valence”.

Return type:

data (pl.DataFrame)

elfen.emotion.get_sentiment_score(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', nan_value: float = 0.0, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the sentiment score of the text.

The sentiment score is calculated as the difference between the number of positive and negative sentiment words divided by the number of tokens. The sentiment score is in the range [-1, 1], where -1 indicates negative sentiment, 0 indicates neutral sentiment, and 1 indicates positive sentiment.

Parameters:

data (pl.DataFrame) – The preprocessed input data. Contains the “nlp” column produced by the NLP backbone.
lexicon (pl.DataFrame) – The sentiment lexicon.
backbone (str) – The NLP backbone to use.
nan_value (float) – The value to use for NaNs. Defaults to 0.0.

Returns:

The input data with the sentiment score column. The column name is “sentiment_score”.

Return type:

data (pl.DataFrame)

elfen.entities module

This module contains functions to extract named entity-related features from text data.

elfen.entities.get_num_entities(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of entities in the text data.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of entities in the text data.

Return type:

data (pl.DataFrame)

elfen.entities.get_num_per_entity_type(data: DataFrame, backbone: str = 'spacy', ent_types: list = ['ORG', 'CARDINAL', 'DATE', 'GPE', 'PERSON', 'MONEY', 'PRODUCT', 'TIME', 'PERCENT', 'WORK_OF_ART', 'QUANTITY', 'NORP', 'LOC', 'EVENT', 'ORDINAL', 'FAC', 'LAW', 'LANGUAGE'], **kwargs: dict[str, str]) → DataFrame

Calculates the number of entities per entity type in the text data.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
ent_types – A list of entity types to calculate the number of entities for. Default is the list of entity types in the spaCy/stanza libraries.

Returns:

A Polars DataFrame containing the number of entities per entity type in the text data.

Return type:

data (pl.DataFrame)

elfen.extractor module

This module contains the Extractor class. The Extractor class is the main class in the ELFEN package and is used to extract features from text data.

class elfen.extractor.Extractor(data: pl.DataFrame, config: dict[str, str] = CONFIG)

Bases: object

The Extractor class is the main class in the ELFEN package and is used to extract features from text data.

The Extractor class takes a Polars DataFrame containing text data and a configuration dictionary as input. The configuration dictionary specifies the text column, the NLP library to use as backbone, the language of the text data, the model to use for processing the text data, and the features to extract.

The Extractor class has methods to extract features from the text data, extract features in a feature group, extract a single feature, get the names of the extracted features, write the extracted features to a CSV file, and clean up the extracted features.

To access the data with the extracted features, use the data attribute of the Extractor class: extractor.data.

extract_feature_group(feature_group: str | list[str] = 'all', feature_area_map: dict[str, list[str]] = FEATURE_AREA_MAP)

Extract all features in a feature group with default settings. Available feature groups are dependency, emotion, entities, information, lexical_richness, morphological, pos, psycholinguistic, readability, semantic, and surface.

NOTE: Multilingual support for features that require lexicons or norms is currently only implemented for emotion/sentiment features. For pscholinguistic features and hedges, only English resources are currently available.

Parameters:

feature_group (str) – The feature group to extract features from.
feature_area_map (dict[str, str]) – A dictionary mapping features to feature areas.

Returns:

The data with the extracted features.

Return type:

data (pl.DataFrame)

extract(features: str | list[str], **kwargs)

Extract a single feature from the data.

Parameters:

features (Union[str, list[str]]) – The feature to extract. Can be a single feature in str format or a list of features.
**kwargs – Additional keyword arguments for the feature extraction. Any lexicons or thresholds required for the feature, or additional parameters the respective feature function takes.

Returns:

None

extract_features() → None

Extracts all features specified in the config.

Returns:: None

get_corpus_frequencies(kind: str = 'token') → dict[str, int]

Get the global frequencies of tokens or lemmas in the data.

Parameters:: kind (str) – The kind of frequency to get. Default is “token”. Options are “token” or “lemma”.
Returns:: A dictionary of the global frequencies of tokens or lemmas in the data.
Return type:: frequencies (dict[str, int])

get_data() → DataFrame

Get the data with the extracted features.

Returns:: The data with the extracted features.
Return type:: data (pl.DataFrame)

get_feature_names() → list[str]

Get the names of the extracted features.

Returns:: A list of the names of the extracted features.
Return type:: feature_names (list[str])

normalize(features: list[str] | str = 'all', **kwargs) → None

Normalize the extracted features to have a mean of 0 and a standard deviation of 1.

Parameters:: features (Union[list[str], str]) – The features to normalize. Default is “all”. Allows for a list of features or a single feature in str format, or ‘all’ to normalize all features.
Returns:: None

ratio_normalize(features: list[str] | str = 'all', ratio: str = 'type', **kwargs) → None

Normalize the extracted features with a specific ratio feature. Ratios available are type, token, and sentence ratios.

Parameters:

features (Union[list[str], str]) – The features to normalize. Default is “all”. Allows for a list of features or a single feature in str format, or ‘all’ to normalize all features.
ratio (str) – The ratio to normalize the features with. Default is “type”.

Returns:

None

remove_constant_cols(): Helper function to remove constant columns from the data. Constant columns are columns with only one unique value.

rescale(features: list[str] | str = 'all', minimum: float = 0, maximum: float = 1, **kwargs) → None

Rescale the extracted features to a specific range.

Parameters:

features (Union[list[str], str]) – The features to rescale. Default is “all”. Allows for a list of features or a single feature in str format, or ‘all’ to rescale all features.
minimum (float) – The minimum value to rescale the features to. Default is 0.
maximum (float) – The maximum value to rescale the features to. Default is 1.

Returns:

None

token_normalize(features: list[str] | str = 'all', **kwargs) → None

Normalize the occurence-based features with the number of tokens.

Parameters:: features (Union[list[str], str]) – The features to normalize. Default is “all”. Allows for a list of features or a single feature in str format, or ‘all’ to normalize all occurence-based features.
Returns:: None

write_csv(filepath: str, **kwargs) → None

Save the extracted features to a CSV file. Cleans up the data by removing helper columns before saving and then writes the data to a CSV using Polars DataFrame to_csv. To specify additional keyword arguments for the to_csv method, pass them as keyword arguments to this function.

Parameters:

filepath (str) – The path to save the CSV file.
**kwargs – Additional keyword arguments for the Polars DataFrame to_csv method.

elfen.features module

This module contains helper mappings to map feature names to the respective functions that calculate them. Additionally, it contains mappings to group features by their respective feature areas.

elfen.information module

This module contains functions for calculating information-theoretic metrics.

The information-theoretic metrics implemented in this module are:

Compressibility:
The ratio of the length of the compressed text to the length of the original text.
Shannon Entropy:
The Shannon entropy of the text data.

elfen.information.entropy(string: str, **kwargs: dict[str, str]) → float

Calculate the Shannon entropy of a string. Helper function for get_entropy.

Parameters:: string (str) – The input string.
Returns:: The Shannon entropy of the input string.
Return type:: entropy (float)

elfen.information.get_compressibility(data: DataFrame, text_column: str = 'text', **kwargs: dict[str, str]) → DataFrame

Calculates the compressibility of the texts in the text column.

The compressibility is the ratio of the length of the compressed text to the length of the original text. This is used as a proxy for the Kolmogorov complexity of the text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
text_column (str) – The name of the column containing the text data.

Returns:

A Polars DataFrame containing the compressibility of the text data. The compressibility is stored in a new column named ‘compressibility’.

Return type:

data (pl.DataFrame)

elfen.information.get_entropy(data: DataFrame, text_column: str = 'text', **kwargs: dict[str, str]) → DataFrame

Calculates the Shannon entropy of the texts in the text column.

The Shannon entropy is a measure of the uncertainty in a random variable.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
text_column (str) – The name of the column containing the text data.

Returns:

A Polars DataFrame containing the Shannon entropy of the text data. The Shannon entropy is stored in a new column named ‘entropy’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness module

This module contains functions to calculate various lexical richness metrics from text data.

The lexical richness metrics implemented in this module are:

Lemma/Token Ratio:
The ratio of the number of lemmas to the number of tokens in the text.
Type/Token Ratio:
The ratio of the number of types to the number of tokens in the text.
Root Type/Token Ratio:
The ratio of the number of types to the square root of the number of tokens in the text.
Corrected Type/Token Ratio:
The ratio of the number of types to the square root of twice the number of tokens in the text.
Herdan’s C:
The logarithm of the number of types divided by the logarithm of the number of tokens.
Summer’s TTR:
The logarithm of the logarithm of the number of types divided by the logarithm of the logarithm of the number of tokens.
Dugast’s Uber Index:
The square of the logarithm of the number of tokens divided by the logarithm of the number of tokens minus the logarithm of the number of types.
Maas’ TTR:
The number of tokens minus the number of types divided by the square of the logarithm of the number of types.
Number of Hapax Legomena:
The number of words that occur only once in the text.
Number of Global Token Hapax Legomena:
The number of words that occur only once in the entire corpus.
Number of Global Lemma Hapax Legomena:
The number of lemmas that occur only once in the entire corpus.
Number of Hapax Dislegomena:
The number of words that occur only once or twice in the text.
Number of Global Token Hapax Dislegomena:
The number of words that occur only once or twice in the entire corpus.
Number of Global Lemma Hapax Dislegomena:
The number of lemmas that occur only once or twice in the entire corpus.
Sichel’s S:
The number of hapax dislegomena divided by the number of types in the text.
Global Sichel’s S:
The number of global token hapax dislegomena divided by the number of types in the text.
Lexical Density:
The ratio of the number of lexical tokens to the number of tokens in the text.
Giroud’s Index:
The number of types divided by the square root of the number of tokens in the text.
Measure of Textual Lexical Diversity (MTLD)
Hypergeometric Distribution Diversity (HD-D)
Moving-Average Type-Token Ratio (MATTR)
Mean Segmental Type-Token Ratio (MSTTR)
Yule’s K
Simpson’s D
Herdan’s Vm

elfen.lexical_richness.get_cttr(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the corrected type-token ratio (CTTR) of a text:

N_types / sqrt(2*N_tokens).

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the corrected type-token ratio of the text data. The corrected type-token ratio is stored in a new column named ‘cttr’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_dugast_u(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the Dougast’s Uber index of a text: log(N_tokens)^2 / (log(N_tokens) - log(N_types)).

Note that the convention is to fill NaNs with 1 as log(1) = 0. For texts with N_types = N_tokens, this will output Inf as division by 0 is not defined.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing Dugast’s Uber index of the text data. Dugast’s Uber index is stored in a new column named ‘dugast_u’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_giroud_index(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the Giroud’s index of a text:

N_types / sqrt(N_tokens).

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing Giroud’s C of the text data. Giroud’s C is stored in a new column named ‘giroud_c’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_global_sichel_s(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the global Sichel’s S of a text: N_global_token_hapax_dislegomena / N_types.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the global Sichel’s S of the text data. The global Sichel’s S is stored in a new column named ‘global_sichel_s’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_hdd(data: DataFrame, backbone: str = 'spacy', draws: int = 42, **kwargs: dict[str, str]) → DataFrame

Calculates the Hypergeometric Distribution Diversity (HD-D) of a text.

The default number of draws is 42. We note, however, that this value should be smaller than the number of tokens in the text and thus will need to be adjusted for most short texts. A number of draws that is too large will result in NaN values.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
draws – The number of draws for the HDD. The default value is 42.

Returns:

A Polars DataFrame containing the HD-D of the text data. The HD-D is stored in a new column named ‘hdd’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_herdan_c(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the Herdan’s C of a text:

log(N_types) / log(N_tokens).

Note that the convention is to fill NaNs with 1 as log(1) = 0.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing Herdan’s C of the text data. Herdan’s C is stored in a new column named ‘herdan_c’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_herdan_v(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the Herdan’s Vm of a text:

Vm^2 = K + (1/N) - (1/V(N)) Vm = sqrt(K + (1/N) - (1/V(N)))

where K is Yule’s K, N is the number of tokens, and V(N) is the number of types:

K = Σ(V(i,N) * (i/n)^2)

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing Herdan’s V of the text data. Herdan’s V is stored in a new column named ‘herdan_v’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_lemma_token_ratio(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the lemma/token ratio of a text:

N_lemmas / N_tokens.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the lemma/token ratio of the text data. The lemma/token ratio is stored in a new column named ‘lemma_token_ratio’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_lexical_density(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the lexical density of a text:

N_lex / N_tokens.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the lexical density of the text data. The lexical density is stored in a new column named ‘lexical_density’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_maas_index(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the Maas’ TTR of a text: (N_tokens - N_types) / log(N_types)^2.

Note that the convention is to fill NaNs with 1 as log(1) = 0.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing Maas’ TTR of the text data. Maas’ TTR is stored in a new column named ‘maas_index’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_mattr(data: DataFrame, backbone: str = 'spacy', window_size: int = 5, **kwargs: dict[str, str]) → DataFrame

Calculates the Moving-Average Type-Token Ratio (MATTR) of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
window_size (int) – The size of the window for the MATTR calculation.

Returns:

A Polars DataFrame containing the MATTR of the text data. The MATTR is stored in a new column named ‘mattr’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_msttr(data: DataFrame, backbone: str = 'spacy', window_size: int = 5, discard: bool = False, **kwargs: dict[str, str]) → DataFrame

Calculates the Mean Segmental Type-Token Ratio (MSTTR) of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
window_size (int) – The size of the window for the MSTTR calculation.
discard (bool) – Whether to discard the last window if it is not complete.

Returns:

A Polars DataFrame containing the MSTTR of the text data. The MSTTR is stored in a new column named ‘msttr’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_mtld(data: DataFrame, threshold: float = 0.72, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the Measure of Textual Lexical Diversity (MTLD) of a text.

For reference https://link.springer.com/article/10.3758/BRM.42.2.381

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
threshold – The threshold value for the MTLD. The default value is 0.72.

Returns:

A Polars DataFrame containing the MTLD of the text data. The MTLD is stored in a new column named ‘mtld’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_n_global_lemma_hapax_dislegomena(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of global hapax dislegomena in a text: lemmas that occur only once or twice in the entire corpus.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of global hapax dislegomena in the text data. The number of global hapax dislegomena is stored in a new column named ‘n_global_lemma_hapax_dislegomena’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_n_global_lemma_hapax_legomena(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of global hapax legomena in a text: lemmas that occur only once in the entire corpus.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of global hapax legomena in the text data. The number of global hapax legomena is stored in a new column named ‘n_global_lemma_hapax_legomena’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_n_global_token_hapax_dislegomena(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of global hapax dislegomena in a text: words that occur only once or twice in the entire corpus.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of global hapax dislegomena in the text data. The number of global hapax dislegomena is stored in a new column named ‘n_global_token_hapax_dislegomena’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_n_global_token_hapax_legomena(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of global hapax legomena in a text: words that occur only once in the entire corpus.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of global hapax legomena in the text data. The number of global hapax legomena is stored in a new column named ‘n_global_hapax_legomena’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_n_hapax_dislegomena(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of hapax dislegomena in a text: words that occur only once or twice.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of hapax dislegomena in the text data. The number of hapax dislegomena is stored in a new column named ‘n_hapax_dislegomena’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_n_hapax_legomena(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of hapax legomena in a text: words that occur only once.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of hapax legomena in the text data. The number of hapax legomena is stored in a new column named ‘n_hapax_legomena’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_rttr(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the root type-token ratio (RTTR) of a text:

N_types / sqrt(N_tokens).

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the root type-token ratio of the text data. The root type-token ratio is stored in a new column named ‘rttr’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_sichel_s(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the Sichel’s S of a text: N_hapax_dislegomena / N_types.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing Sichel’s S of the text data. Sichel’s S is stored in a new column named ‘sichel_s’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_simpsons_d(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the Simpson’s D of a text.

D = Σ(V(i,N) * (i/n) * ((i-1)/(n-1)))

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing Simpson’s D of the text data. Simpson’s D is stored in a new column named ‘simpsons_d’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_summer_index(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the Summer’s TTR of a text:

log(log(N_types)) / log(log(N_tokens)).

Note that the convention is to fill NaNs with 1 as log(1) = 0.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the Summer’s TTR of the text data. Summer’s TTR is stored in a new column named ‘summer_index’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_ttr(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the type-token ratio (TTR) of a text:

N_types / N_tokens.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the type-token ratio of the text data. The type-token ratio is stored in a new column named ‘ttr’.

Return type:

data (pl.DataFrame)

elfen.lexical_richness.get_yule_k(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the Yule’s K of a text. Yule’s K is a characteristic of the vocabulary richness of a text. It is calculated as:

K = 10^4 * (Σ(V(i,N) * (i/n)^2) - n) / n^2 = 10^4 * ((Σ(V(i,N) * (i^2/n^4)) - (1/n))

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data.

Returns:

A Polars DataFrame containing Yule’s K of the text data. Yule’s K is stored in a new column named ‘yule_k’.

Return type:

data (pl.DataFrame)

elfen.morphological module

This module contains functions to extract morphological features of text data.

The morphological features implemented in this module are:

Number of tokens with a specific morphological feature: The number of
tokens that have a specific morphological feature, such as VerbForm, Number, etc.

elfen.morphological.get_morph_feats(data: pl.DataFrame, backbone: str = 'spacy', morph_config: dict[str, str] = MORPH_CONFIG) → pl.DataFrame

Extracts morphological features from the text data.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
morph_config (dict[str, str]) – A dictionary containing the configuration for extracting morphological features. The keys are POS, and the values are dictionaries containing names of features as keys and their configurations as values. The configuration should follow the format as in the MORPH_CONFIG dictionary in elfen/configs/morphological_config.py.

Returns:

A Polars DataFrame containing the morphological features of the text data.

Return type:

data (pl.DataFrame)

elfen.pos module

This module contains functions to calculate various part-of-speech (POS)- related features from text data.

The POS-related features implemented in this module are:

Number of lexical tokens: The number of tokens that are nouns, verbs,
adjectives, or adverbs.
POS variability: The ratio of the number of unique POS tags to the
number of tokens.
Number of tokens per POS tag: The number of tokens for each POS tag.

elfen.pos.get_num_lexical_tokens(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of lexical tokens in the text.

Lexical tokens are tokens that are nouns, verbs, adjectives, or adverbs.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of lexical tokens in the text data. The number of lexical tokens is stored in a new column named ‘n_lexical_tokens’.

Return type:

data (pl.DataFrame)

elfen.pos.get_num_per_pos(data: DataFrame, backbone: str = 'spacy', pos_tags: list = ['ADJ', 'ADP', 'ADV', 'AUX', 'CONJ', 'CCONJ', 'DET', 'INTJ', 'NOUN', 'NUM', 'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X'], **kwargs: dict[str, str]) → DataFrame

Calculates the number of tokens per part-of-speech tag in the text data.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
pos_tags – A list of part-of-speech tags to calculate the number of tokens for. Default is the Universal POS tagset.

Returns:

A Polars DataFrame containing the number of tokens per part-of-speech tag in the text data. The number of tokens per part-of-speech tag is stored in new columns named ‘n_{pos}’ where {pos} is the part-of speech tag.

Return type:

data (pl.DataFrame)

elfen.pos.get_pos_variability(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the variability of part-of-speech tags in the text data.

The variability is the ratio of the number of unique part-of-speech tags to the number of tokens in the text data.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the variability of part-of-speech tags in the text data. The variability is stored in a new column named ‘pos_variability’.

Return type:

data (pl.DataFrame)

elfen.preprocess module

This module contains functions to preprocess text data.

This preprocessing enables the fast extraction of features from the text data.

The preprocessing steps implemented in this module are:

Tokenization:
The text data is tokenized into individual words.
Lemmatization:
The words in the text data are converted to their base form.
Syllabification:
The words in the text data are split into syllables.
POS Tagging:
The words in the text data are tagged with their part-of-speech.
Dependency Parsing:
The words in the text data are parsed for dependencies.
Named Entity Recognition:
The named entities in the text data are identified.

elfen.preprocess.get_lemmas(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Gets the lemmas of the text data.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the lemmas of the text data. The lemmas are stored in a new column named ‘lemmas’.

Return type:

data (pl.DataFrame)

elfen.preprocess.get_tokens(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Gets the tokens of the text data.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the tokens of the text data. The tokens are stored in a new column named ‘tokens’.

Return type:

data (pl.DataFrame)

elfen.preprocess.preprocess_data(data: DataFrame, text_column: str = 'text', backbone: str = 'spacy', model: str = 'en_core_web_sm', max_length: int = 1000000, batch_size: int = 1, n_process: int = 1, **kwargs: dict[str, str]) → DataFrame

Preprocesses the text data using the specified NLP library.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
text_column – The name of the column containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
model (str) – The name of the model used by the NLP library.
max_length (int) – The maximum number of characters to process.

Returns:

A Polars DataFrame containing the processed text data. The processed data is stored in a new column named ‘nlp’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic module

This module contains functions for calculating psycholinguistic features from text data.

If you are using this module, please cite the respective sources for the norms used in the functions. Consult them for more information on the norms, their collection, and their interpretation in your analyses.

The psycholinguistic features implemented in this module are:

Abstractness/Concreteness:
- Average concreteness score
- Average standard deviation of the concreteness score
- Number of low concreteness words
- Number of high concreteness words
- Number of controversial concreteness words
- Minimum concreteness score
- Maximum concreteness score
- Standard deviation of concreteness scores
Age of Acquisition:
- Average age of acquisition score
- Average standard deviation of the age of acquisition score
- Number of low age of acquisition words
- Number of high age of acquisition words
- Number of controversial age of acquisition words
- Minimum age of acquisition score
- Maximum age of acquisition score
- Standard deviation of age of acquisition scores
Word Prevalence:
- Average prevalence score
- Number of low prevalence words
- Number of high prevalence words
- Minimum prevalence score
- Maximum prevalence score
- Standard deviation of prevalence scores
Socialness:
- Average socialness score
- Average standard deviation of the socialness score
- Number of low socialness words
- Number of high socialness words
- Number of controversial socialness words
- Minimum socialness score
- Maximum socialness score
- Standard deviation of socialness scores
Iconicity:
- Average iconicity score
- Average standard deviation of the iconicity score
- Number of low iconicity words
- Number of high iconicity words
- Number of controversial iconicity words
- Minimum iconicity score
- Maximum iconicity score
- Standard deviation of iconicity scores
Sensorimotor; per dimension (e.g., Auditory, Gustatory, Haptic, etc.):
- Average sensorimotor score
- Average standard deviation of the sensorimotor score
- Number of low sensorimotor words
- Number of high sensorimotor words
- Number of controversial sensorimotor words
- Minimum sensorimotor score
- Maximum sensorimotor score
- Standard deviation of sensorimotor scores

elfen.psycholinguistic.filter_concreteness_norms(concreness_norms: DataFrame, words: list, **kwargs: dict[str, str]) → DataFrame

Filters the concreteness norms dataset by a list of words.

Parameters:

concreness_norms – A Polars DataFrame containing the concreteness norms.
words (list[str]) – A list of words to filter the concreteness norms dataset.

Returns:

A Polars DataFrame containing the filtered concreteness norms dataset.

Return type:

filtered_concreteness_norms (pl.DataFrame)

elfen.psycholinguistic.get_avg_aoa(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the average age of acquisition score of a text. NaN/Null values indicate that no word in the text was found in the age of acquisition norms.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the age of acquisition norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the average age of acquisition score of the text data. The average age of acquisition score is stored in a new column named ‘avg_aoa’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_avg_concreteness(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the average concreteness score of a text. NaN/Null values indicate that no word in the text was found in the concreteness norms.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the concreteness norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the average concreteness score of the text data. The average concreteness score is stored in a new column named ‘avg_concreteness’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_avg_iconicity(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the average iconicity score of a text. NaN/Null values indicate that no word in the text was found in the iconicity norms.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the iconicity norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the average iconicity score of the text data. The average iconicity score is stored in a new column named ‘avg_iconicity’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_avg_prevalence(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the average prevalence score of a text. NaN/Null values indicate that no word in the text was found in the prevalence norms.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
prevalence_norms (pl.DataFrame) – A Polars DataFrame containing the word prevalence norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the average prevalence score of the text data. The average prevalence score is stored in a new column named ‘avg_prevalence’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_avg_sd_aoa(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the average standard deviation of age of acquisition score of a text. NaN/Null values indicate that no word in the text was found in the age of acquisition norms.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the age of acquisition norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the average standard deviation of age of acquisition score of the text data. The average standard deviation of age of acquisition score is stored in a new column named ‘avg_sd_aoa’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_avg_sd_concreteness(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the average standard deviation of concreteness score of a text. NaN/Null values indicate that no word in the text was found in the concreteness norms.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the concreteness norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the average standard deviation of concreteness score of the text data. The average standard deviation of concreteness score is stored in a new column named ‘avg_sd_concreteness’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_avg_sd_iconicity(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the average standard deviation of iconicity score of a text. NaN/Null values indicate that no word in the text was found in the iconicity norms.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the iconicity norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the average standard deviation of iconicity score of the text data. The average standard deviation of iconicity score is stored in a new column named ‘avg_sd_iconicity’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_avg_sd_sensorimotor(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', sensorimotor_vars: list[str] = {'en': ['Auditory', 'Gustatory', 'Haptic', 'Interoceptive', 'Olfactory', 'Visual', 'Foot_leg', 'Hand_arm', 'Head', 'Mouth', 'Torso'], 'it': ['Auditory', 'Gustatory', 'Haptic', 'Olfactory', 'Visual']}, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the average standard deviation of sensorimotor variable score of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the sensorimotor norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
sensorimotor_vars (list[str]) – A list of sensorimotor variables to calculate the average standard deviation for.

Returns:

A Polars DataFrame containing the average: standard deviation of sensorimotor variable score of the text data. The average standard deviation of sensorimotor variable score is stored in new columns named ‘avg_sd_{var}’ where {var} is the sensorimotor variable.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_avg_sd_socialness(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the average standard deviation of socialness score of a text. NaN/Null values indicate that no word in the text was found in the socialness norms.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the socialness norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the average standard deviation of socialness score of the text data. The average standard deviation of socialness score is stored in a new column named ‘avg_sd_socialness’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_avg_sensorimotor(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', sensorimotor_vars: list[str] = {'en': ['Auditory', 'Gustatory', 'Haptic', 'Interoceptive', 'Olfactory', 'Visual', 'Foot_leg', 'Hand_arm', 'Head', 'Mouth', 'Torso'], 'it': ['Auditory', 'Gustatory', 'Haptic', 'Olfactory', 'Visual']}, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the average sensorimotor variable score of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the sensorimotor norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
sensorimotor_vars (list[str]) – A list of sensorimotor variables to calculate the average score for. Defaults to SENSORIMOTOR_VARS.

Returns:

A Polars DataFrame containing the average sensorimotor score of the text data. The average sensorimotor score is stored in a new column named ‘avg_sensorimotor’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_avg_socialness(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the average socialness score of a text. NaN/Null values indicate that no word in the text was found in the socialness norms.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the socialness norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the average socialness score of the text data. The average socialness score is stored in a new column named ‘avg_socialness’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_max_aoa(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the maximum age of acquisition score of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the age of acquisition norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the maximum age of acquisition score of the text data. The maximum age of acquisition score is stored in a new column named ‘max_aoa’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_max_concreteness(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the maximum concreteness score of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the concreteness norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the maximum concreteness score of the text data. The maximum concreteness score is stored in a new column named ‘max_concreteness’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_max_iconicity(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the maximum iconicity score of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the iconicity norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the maximum iconicity score of the text data. The maximum iconicity score is stored in a new column named ‘max_iconicity’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_max_prevalence(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the maximum prevalence score of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the word prevalence norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the maximum prevalence score of the text data. The maximum prevalence score is stored in a new column named ‘max_prevalence’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_max_sensorimotor(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', sensorimotor_vars: list[str] = {'en': ['Auditory', 'Gustatory', 'Haptic', 'Interoceptive', 'Olfactory', 'Visual', 'Foot_leg', 'Hand_arm', 'Head', 'Mouth', 'Torso'], 'it': ['Auditory', 'Gustatory', 'Haptic', 'Olfactory', 'Visual']}, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the maximum sensorimotor variable score of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the sensorimotor norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
sensorimotor_vars (list[str]) – A list of sensorimotor variables to calculate the maximum score for. Defaults to SENSORIMOTOR_VARS.
language (str) – The language of the text data. Defaults to “en”.

Returns:

A Polars DataFrame containing the maximum sensorimotor score of the text data. The maximum sensorimotor score is stored in new columns named ‘max_{var}’ where {var} is the sensorimotor variable.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_max_socialness(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the maximum socialness score of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the socialness norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the maximum socialness score of the text data. The maximum socialness score is stored in a new column named ‘max_socialness’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_min_aoa(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the minimum age of acquisition score of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the age of acquisition norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the minimum age of acquisition score of the text data. The minimum age of acquisition score is stored in a new column named ‘min_aoa’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_min_concreteness(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the minimum concreteness score of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the concreteness norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the minimum concreteness score of the text data. The minimum concreteness score is stored in a new column named ‘min_concreteness’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_min_iconicity(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the minimum iconicity score of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the iconicity norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the minimum iconicity score of the text data. The minimum iconicity score is stored in a new column named ‘min_iconicity’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_min_prevalence(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the minimum prevalence score of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the word prevalence norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the minimum prevalence score of the text data. The minimum prevalence score is stored in a new column named ‘min_prevalence’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_min_sensorimotor(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', sensorimotor_vars: list[str] = {'en': ['Auditory', 'Gustatory', 'Haptic', 'Interoceptive', 'Olfactory', 'Visual', 'Foot_leg', 'Hand_arm', 'Head', 'Mouth', 'Torso'], 'it': ['Auditory', 'Gustatory', 'Haptic', 'Olfactory', 'Visual']}, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the minimum sensorimotor variable score of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the sensorimotor norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
sensorimotor_vars (list[str]) – A list of sensorimotor variables to calculate the minimum score for. Defaults to SENSORIMOTOR_VARS.
language (str) – The language of the text data. Defaults to “en”.

Returns:

A Polars DataFrame containing the minimum sensorimotor score of the text data. The minimum sensorimotor score is stored in new columns named ‘min_{var}’ where {var} is the sensorimotor variable.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_min_socialness(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the minimum socialness score of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the socialness norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the minimum socialness score of the text data. The minimum socialness score is stored in a new column named ‘min_socialness’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_n_controversial_aoa(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', threshold: float = 4.5, **kwargs: dict[str, str]) → DataFrame

Calculates the number of controversial age of acquisition words in a text (i.e. items with a high standard deviation in the ratings).

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the age of acquisition norms.
threshold (float) – The threshold for the standard deviation. Defaults to 4.5.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of controversial age of acquisition words in the text data. The number of controversial age of acquisition words is stored in a new column named ‘n_controversial_aoa’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_n_controversial_concreteness(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', threshold: float = 2.0, **kwargs: dict[str, str]) → DataFrame

Calculates the number of controversial concreteness words in a text (i.e. items with a high standard deviation in the ratings).

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the concreteness norms.
threshold (float) –

The threshold for the standard deviation.
Defaults to 2.5.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of controversial concreteness words in the text data. The number of controversial concreteness words is stored in a new column named ‘n_controversial_concreteness’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_n_controversial_iconicity(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', threshold: float = 2.5, **kwargs: dict[str, str]) → DataFrame

Calculates the number of controversial iconicity words in a text (i.e. items with a high standard deviation in the ratings).

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the iconicity norms.
threshold (float) – The threshold for the standard deviation. Defaults to 2.5.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of controversial iconicity words in the text data. The number of controversial iconicity words is stored in a new column named ‘n_controversial_iconicity’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_n_controversial_sensorimotor(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', threshold: float = 2.0, sensorimotor_vars: list[str] = {'en': ['Auditory', 'Gustatory', 'Haptic', 'Interoceptive', 'Olfactory', 'Visual', 'Foot_leg', 'Hand_arm', 'Head', 'Mouth', 'Torso'], 'it': ['Auditory', 'Gustatory', 'Haptic', 'Olfactory', 'Visual']}, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the number of controversial sensorimotor words in a text (i.e. items with a high standard deviation in the ratings).

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the sensorimotor norms.
threshold (float) – The threshold for the standard deviation. Defaults to 2.0.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
sensorimotor_vars (list[str]) – A list of sensorimotor variables to calculate the number of controversial words for.

Returns:

A Polars DataFrame containing the number of controversial sensorimotor words in the text data. The number of controversial sensorimotor words is stored in new columns named ‘n_controversial_{var}’ where {var} is the sensorimotor variable.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_n_controversial_socialness(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', threshold: float = 2.0, **kwargs: dict[str, str]) → DataFrame

Calculates the number of controversial socialness words in a text (i.e. items with a high standard deviation in the ratings).

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the socialness norms.
threshold (float) – The threshold for the standard deviation. Defaults to 2.0.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of controversial socialness words in the text data. The number of controversial socialness words is stored in a new column named ‘n_controversial_socialness’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_n_high_aoa(data: DataFrame, lexicon: DataFrame, threshold: float = 10.0, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of high age of acquisition words in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the age of acquisition norms.
threshold (float) – The threshold for the high age of acquisition words. Defaults to 10.0.

Returns:

A Polars DataFrame containing the number of high age of acquisition words in the text data. The number of high age of acquisition words is stored in a new column named ‘n_high_aoa’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_n_high_concreteness(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', threshold: float = 3.33, **kwargs: dict[str, str]) → DataFrame

Calculates the number of high concreteness words in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the concreteness norms.
threshold (float) – The threshold for the high concreteness words. Defaults to 3.33.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of high concreteness words in the text data. The number of high concreteness words is stored in a new column named ‘n_high_concreteness’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_n_high_iconicity(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', threshold: float = 3.66, **kwargs: dict[str, str]) → DataFrame

Calculates the number of high iconicity words in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the iconicity norms.
threshold (float) – The threshold for the high iconicity words. Defaults to 3.66.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of high iconicity words in the text data. The number of high iconicity words is stored in a new column named ‘n_high_iconicity’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_n_high_prevalence(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', threshold: float = 1.0, **kwargs: dict[str, str]) → DataFrame

Calculate the number of high prevalence words in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicons – A Polars DataFrame containing the word prevalence norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
threshold (float) – The threshold for the high prevalence words. Defaults to 1.0.

Returns:

A Polars DataFrame containing the number of high prevalence words in the text data. The number of high prevalence words is stored in a new column named ‘n_high_prevalence’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_n_high_sensorimotor(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', threshold: float = 3.66, sensorimotor_vars: list[str] = {'en': ['Auditory', 'Gustatory', 'Haptic', 'Interoceptive', 'Olfactory', 'Visual', 'Foot_leg', 'Hand_arm', 'Head', 'Mouth', 'Torso'], 'it': ['Auditory', 'Gustatory', 'Haptic', 'Olfactory', 'Visual']}, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the number of high-rating sensorimotor words in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the sensorimotor norms.
threshold (float) – The threshold for the high-rating sensorimotor words. Defaults to 3.66.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
sensorimotor_vars (list[str]) – A list of sensorimotor variables to calculate the number of high-rating words for. Defaults to SENSORIMOTOR_VARS.

Returns:

A Polars DataFrame containing the number of high-rating sensorimotor words in the text data. The number of high-rating sensorimotor words is stored in new columns named ‘n_high_{var}’ where {var} is the sensorimotor variable.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_n_high_socialness(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', threshold: float = 3.66, **kwargs: dict[str, str]) → DataFrame

Calculates the number of high socialness words in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the socialness norms.
threshold (float) – The threshold for the high socialness words. Defaults to 3.66.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of high socialness words in the text data. The number of high socialness words is stored in a new column named ‘n_high_socialness’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_n_low_aoa(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', threshold: float = 10.0, **kwargs: dict[str, str]) → DataFrame

Calculates the number of low age of acquisition words in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
aoa_norms (pl.DataFrame) – A Polars DataFrame containing the age of acquisition norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
threshold (float) – The threshold for the low age of acquisition words. Defaults to 10.0.

Returns:

A Polars DataFrame containing the number of low age of acquisition words in the text data. The number of low age of acquisition words is stored in a new column named ‘n_low_aoa’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_n_low_concreteness(data: DataFrame, lexicon: DataFrame, threshold: float = 1.66, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of low concreteness words in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the concreteness norms.
threshold (float) – The threshold for the low concreteness words. Defaults to 1.66.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of low concreteness words in the text data. The number of low concreteness words is stored in a new column named ‘n_low_concreteness’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_n_low_iconicity(data: DataFrame, lexicon: DataFrame, threshold: float = 2.33, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of low iconicity words in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the iconicity norms.
threshold (float) – The threshold for the low iconicity words. Defaults to 2.33.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of low iconicity words in the text data. The number of low iconicity words is stored in a new column named ‘n_low_iconicity’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_n_low_prevalence(data: DataFrame, lexicon: DataFrame, threshold: float = 1.0, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculate the number of low prevalence words in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the word prevalence norms.

Returns:

A Polars DataFrame containing the number of low prevalence words in the text data. The number of low prevalence words is stored in a new column named ‘n_low_prevalence’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_n_low_sensorimotor(data: DataFrame, lexicon: DataFrame, threshold: float = 2.33, backbone: str = 'spacy', sensorimotor_vars: list[str] = {'en': ['Auditory', 'Gustatory', 'Haptic', 'Interoceptive', 'Olfactory', 'Visual', 'Foot_leg', 'Hand_arm', 'Head', 'Mouth', 'Torso'], 'it': ['Auditory', 'Gustatory', 'Haptic', 'Olfactory', 'Visual']}, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the number of low-rating sensorimotor words in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the sensorimotor norms.
threshold (float) – The threshold for the low-rating sensorimotor words. Defaults to 2.33.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
sensorimotor_vars (list[str]) – A list of sensorimotor variables to calculate the number of low-rating words for. Defaults to SENSORIMOTOR_VARS.

Returns:

A Polars DataFrame containing the number of low-rating sensorimotor words in the text data. The number of low-rating sensorimotor words is stored in new columns named ‘n_low_{var}’ where {var} is the sensorimotor variable.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_n_low_socialness(data: DataFrame, lexicon: DataFrame, threshold: float = 2.33, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of low socialness words in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the socialness norms.
threshold (float) – The threshold for the low socialness words. Defaults to 1.66.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of low socialness words in the text data. The number of low socialness words is stored in a new column named ‘n_low_socialness’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_sd_aoa(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the standard deviation of age of acquisition scores in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the age of acquisition norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the standard deviation of age of acquisition scores in the text data. The standard deviation of age of acquisition scores is stored in a new column named ‘sd_aoa’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_sd_concreteness(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the standard deviation of concreteness scores in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the concreteness norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the standard deviation of concreteness scores in the text data. The standard deviation of concreteness scores is stored in a new column named ‘sd_concreteness’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_sd_iconicity(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the standard deviation of iconicity scores in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the iconicity norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the standard deviation of iconicity scores in the text data. The standard deviation of iconicity scores is stored in a new column named ‘sd_iconicity’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_sd_prevalence(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the standard deviation of prevalence scores in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the word prevalence norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the standard deviation of prevalence scores in the text data. The standard deviation of prevalence scores is stored in a new column named ‘sd_prevalence’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_sd_sensorimotor(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', sensorimotor_vars: list[str] = {'en': ['Auditory', 'Gustatory', 'Haptic', 'Interoceptive', 'Olfactory', 'Visual', 'Foot_leg', 'Hand_arm', 'Head', 'Mouth', 'Torso'], 'it': ['Auditory', 'Gustatory', 'Haptic', 'Olfactory', 'Visual']}, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Calculates the standard deviation of sensorimotor variable scores in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the sensorimotor norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
sensorimotor_vars (list[str]) – A list of sensorimotor variables to calculate the standard deviation for. Defaults to SENSORIMOTOR_VARS.
language (str) – The language of the text data. Defaults to “en”.

Returns:

A Polars DataFrame containing the standard deviation of sensorimotor scores in the text data. The standard deviation of sensorimotor scores is stored in new columns named ‘sd_{var}’ where {var} is the sensorimotor variable.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.get_sd_socialness(data: DataFrame, lexicon: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the standard deviation of socialness scores in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
lexicon (pl.DataFrame) – A Polars DataFrame containing the socialness norms.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the standard deviation of socialness scores in the text data. The standard deviation of socialness scores is stored in a new column named ‘sd_socialness’.

Return type:

data (pl.DataFrame)

elfen.psycholinguistic.load_aoa_norms(path: str, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Loads the age of acquisition norms dataset.

Parameters:: path (str) – The path to the age of acquisition norms dataset.
Returns:: A Polars DataFrame containing the age of acquisition norms dataset.
Return type:: aoa_norms (pl.DataFrame)

elfen.psycholinguistic.load_concreteness_norms(path: str, language: str = 'en', **kwargs: dict[str, str]) → DataFrame

Loads the concreteness norms dataset.

Parameters:: path (str) – The path to the concreteness norms dataset.
Returns:: A Polars DataFrame containing the concreteness norms dataset.
Return type:: concreteness_norms (pl.DataFrame)

elfen.psycholinguistic.load_iconicity_norms(path: str, **kwargs: dict[str, str]) → DataFrame

Loads the iconicity norms dataset.

Parameters:: path (str) – The path to the iconicity norms dataset.
Returns:: A Polars DataFrame containing the iconicity norms dataset.
Return type:: iconicity_norms (pl.DataFrame)

elfen.psycholinguistic.load_prevalence_norms(path: str, **kwargs: dict[str, str]) → DataFrame

Loads the word prevalence norms dataset.

Parameters:: path (str) – The path to the word prevalence norms dataset.
Returns:: A Polars DataFrame containing the word prevalence norms dataset.
Return type:: prevalence_norms (pl.DataFrame)

elfen.psycholinguistic.load_sensorimotor_norms(path: str, language: str = 'en', sensorimotor_vars: list[str] = {'en': ['Auditory', 'Gustatory', 'Haptic', 'Interoceptive', 'Olfactory', 'Visual', 'Foot_leg', 'Hand_arm', 'Head', 'Mouth', 'Torso'], 'it': ['Auditory', 'Gustatory', 'Haptic', 'Olfactory', 'Visual']}, **kwargs: dict[str, str]) → DataFrame

Loads the sensorimotor norms dataset.

Parameters:: path (str) – The path to the sensorimotor norms dataset.
Returns:: A Polars DataFrame containing the sensorimotor norms dataset.
Return type:: sensorimotor_norms (pl.DataFrame)

elfen.psycholinguistic.load_socialness_norms(path: str, **kwargs: dict[str, str]) → DataFrame

Loads the socialness norms dataset.

Parameters:: path (str) – The path to the socialness norms dataset.
Returns:: A Polars DataFrame containing the socialness norms dataset.
Return type:: socialness_norms (pl.DataFrame)

elfen.ratios module

elfen.ratios.get_feature_sentence_ratio(data: DataFrame, features: list[str], backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Gets the ratio of given features to the total number of sentences.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
features (list[str]) – A list of features to calculate the ratio for. Note that the features should be present in the data as column names.

Returns:

A Polars DataFrame containing the ratio of the given features to the total number of sentences. The ratios are stored in new columns with the feature names suffixed by ‘_sentence_ratio’.

Return type:

data (pl.DataFrame)

elfen.ratios.get_feature_token_ratio(data: DataFrame, features: list[str], backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Gets the ratio of given features to the total number of tokens.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
features (list[str]) – A list of features to calculate the ratio for. Note that the features should be present in the data as column names.

Returns:

A Polars DataFrame containing the ratio of the given features to the total number of tokens. The ratios are stored in new columns with the feature names suffixed by ‘_token_ratio’.

Return type:

data (pl.DataFrame)

elfen.ratios.get_feature_type_ratio(data: DataFrame, features: list[str], backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Gets the ratio of given features to the total number of types.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
features (list[str]) – A list of features to calculate the ratio for. Note that the features should be present in the data as column names.

Returns:

A Polars DataFrame containing the ratio of the given features to the total number of types. The ratios are stored in new columns with the feature names suffixed by ‘_type_ratio’.

Return type:

data (pl.DataFrame)

elfen.readability module

This module contains functions to calculate readability scores from text data. Readability scores are used to assess the readability, i.e., the complexity, of a text.

The readability scores implemented in this module are:

Total number of syllables
Number of monosyllables
Number of polysyllables
Flesch Reading Ease
Flesch-Kincaid Grade Level
Automated Readability Index (ARI)
Simple Measure of Gobbledygook (SMOG)
Coleman-Liau Index (CLI)
Gunning Fog Index
LIX
RIX

elfen.readability.get_ari(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the Automated Readability Index (ARI) of a text.

Parameters:

data (pl.DataFrame) – APolars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’. Not supported for Stanza backbone.

Returns:

A Polars DataFrame containing the Automated Readability Index of the text data. The Automated Readability Index is stored in a new column named ‘ari’.

Return type:

data (pl.DataFrame)

elfen.readability.get_cli(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the Coleman-Liau Index (CLI) of a text.

Parameters:

data (pl.DataFrame) – APolars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’. Not supported for Stanza backbone.

Returns:

A Polars DataFrame containing the Coleman-Liau Index of the text data. The Coleman-Liau Index is stored in a new column named ‘cli’.

Return type:

data (pl.DataFrame)

elfen.readability.get_flesch_kincaid_grade(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the Flesch-Kincaid Grade Level of a text.

Parameters:

data (pl.DataFrame) – APolars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’. Not supported for Stanza backbone.

Returns:

A Polars DataFrame containing the Flesch-Kincaid Grade Level of the text data. The Flesch-Kincaid Grade Level is stored in a new column named ‘flesch_kincaid_grade’.

Return type:

data (pl.DataFrame)

elfen.readability.get_flesch_reading_ease(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the Flesch Reading Ease score of a text.

Parameters:

data (pl.DataFrame) – APolars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’. Not supported for Stanza backbone.

Returns:

A Polars DataFrame containing the Flesch Reading Ease score of the text data. The Flesch Reading Ease score is stored in a new column named ‘flesch_reading_ease’.

Return type:

data (pl.DataFrame)

elfen.readability.get_gunning_fog(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the Gunning Fog Index of a text.

Parameters:

data (pl.DataFrame) – APolars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’. Not supported for Stanza backbone.

Returns:

A Polars DataFrame containing the Gunning Fog Index of the text data. The Gunning Fog Index is stored in a new column named ‘gunning_fog’.

Return type:

data (pl.DataFrame)

elfen.readability.get_lix(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the LIX of a text.

Parameters:

data (pl.DataFrame) – APolars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the LIX of the text data. The LIX is stored in a new column named ‘lix’.

Return type:

data (pl.DataFrame)

elfen.readability.get_num_monosyllables(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of monosyllables in a text.

Monosyllables are words with one syllable.

Parameters:

data (pl.DataFrame) – APolars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’. Not supported for Stanza backbone.

Returns:

A Polars DataFrame containing the number of monosyllables in the text data. The number of monosyllables is stored in a new column named ‘n_monosyllables’.

Return type:

data (pl.DataFrame)

elfen.readability.get_num_polysyllables(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of polysyllables in a text.

Polysyllables are words with three or more syllables.

Parameters:

data (pl.DataFrame) – APolars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’. Not supported for Stanza backbone.

Returns:

A Polars DataFrame containing the number of polysyllables in the text data. The number of polysyllables is stored in a new column named ‘n_polysyllables’.

Return type:

data (pl.DataFrame)

elfen.readability.get_num_syllables(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of syllables in a text.

Syllables are calculated using the syllables_count attribute of the tokens in the text data.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’. Not supported for Stanza backbone.

Returns:

A Polars DataFrame containing the number of syllables in the text data. The number of syllables is stored in a new column named ‘n_syllables’.

Return type:

data (pl.DataFrame)

elfen.readability.get_rix(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the RIX of a text.

Parameters:

data (pl.DataFrame) – APolars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’. Not supported for Stanza backbone.

Returns:

A Polars DataFrame containing the RIX of the text data. The RIX is stored in a new column named ‘rix’.

Return type:

data (pl.DataFrame)

elfen.readability.get_smog(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the Simple Measure of Gobbledygook (SMOG) of a text.

Parameters:

data (pl.DataFrame) – APolars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’. Not supported for Stanza backbone.

Returns:

A Polars DataFrame containing the Simple Measure of Gobbledygook of the text data. The Simple Measure of Gobbledygook is stored in a new column named ‘smog’.

Return type:

data (pl.DataFrame)

elfen.resources module

This module contains functions to download external resources.

If you are using the resources for research, please cite the original authors.

elfen.resources.download_lexicon(link: str, path: str, filename: str | None = None) → None

Download a lexicon from a link and save it to a path.

Parameters:

link (str) – Link to the lexicon.
path (str) – Path to save the lexicon.
filename (str) – Name of the file to save the lexicon.

Returns:

None

elfen.resources.get_bibtex() → str

Print the bibtex citation for all the resources in the RESOURCE_MAP, and the package itself.

The citation keys for the lexicons are the names of the lexicons in the RESOURCE_MAP. The citation key for the package is “maurer-2024-elfen”.

Parameters:: None
Returns:: Bibtex citation for all the resources.
Return type:: bibxtex (str)

elfen.resources.get_resource(feature: str) → None

Download a resource from the RESOURCE_MAP.

Parameters:: feature (str) – Name of the feature to download.
Returns:: None

elfen.resources.list_external_resources() → None

List all the external resources available in the RESOURCE_MAP.

Parameters:: None
Returns:: None

elfen.schemas module

This module contains schema definitions for the different external resources. These schemas are used in Polars to enable type-safe and fast data manipulation.

elfen.semantic module

This module provides functions to extract semantic features from text.

The WordNet features are collected using the wn package. It uses Open Multilingual WordNet to provide information about words in different languages.

If you are using the resources for research, please cite the original authors of Open Multilingual WordNet.

The semantic features implemented in this module are: - Number of hedge words - Ratio of hedge words - Average number of synsets - Average number of synsets per POS tag - Number of words with a low number of synsets - Number of words with a high number of synsets - Number of words with a low number of synsets per POS tag - Number of words with a high number of synsets per POS tag

elfen.semantic.get_avg_num_synsets(data: DataFrame, backbone: str = 'spacy', language: str = 'en', pos_tags: list[str] = ['NOUN', 'VERB', 'ADJ', 'ADV'], **kwargs: dict[str, str]) → DataFrame

Calculates the average number of synsets in a text.

WordNet synsets serve as a proxy for the ambiguity/polysemy of a word.

Parameters:

data (pl.DataFrame) – Polars DataFrame.
backbone (str) – NLP library used. ‘spacy’ or ‘stanza’.
language (str) – Language of the text. Defaults to English (‘en’).
pos_tags (list[str]) – List of POS tags to consider. Defaults to lexical tokens.

Returns:

Polars DataFrame with the average number of synsets. The column is named ‘avg_n_synsets’.

Return type:

data (pl.DataFrame)

elfen.semantic.get_avg_num_synsets_per_pos(data: DataFrame, backbone: str = 'spacy', language: str = 'en', pos_tags: list[str] = ['NOUN', 'VERB', 'ADJ', 'ADV'], nan_value: float = 0, **kwargs: dict[str, str]) → DataFrame

Calculates the average number of synsets per POS in a text.

Parameters:

data (pl.DataFrame) – Polars DataFrame.
backbone (str) – NLP library used. ‘spacy’ or ‘stanza’.
language (str) – Language of the text. Defaults to English (‘en’).
pos_tags (list[str]) – List of POS tags to consider. Defaults to lexical tokens.
nan_value – Value to fill NaNs with. Defaults to 0.

Returns:

Polars DataFrame with the average number of synsets per POS. The columns are named ‘avg_n_synsets_{pos}’.

Return type:

data (pl.DataFrame)

elfen.semantic.get_hedges_ratio(data: DataFrame, lexicon: list[str], text_column: str = 'text', backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the ratio of hedges in a text.

Parameters:

data (pl.DataFrame) – Polars DataFrame.
hedges (list[str]) – List of hedges.
text_column (str) – Name of the column containing the text.
backbone (str) – NLP library used.

Returns:

Polars DataFrame with the hedges ratio.

Return type:

data (pl.DataFrame)

elfen.semantic.get_high_synsets_per_pos(data: DataFrame, backbone: str = 'spacy', language: str = 'en', pos_tags: list[str] = ['NOUN', 'VERB', 'ADJ', 'ADV'], threshold: int = 5, **kwargs: dict[str, str]) → DataFrame

Calculates the number of words with a high number of synsets per POS in a text.

Parameters:

data (pl.DataFrame) – Polars DataFrame.
backbone (str) – NLP library used. ‘spacy’ or ‘stanza’.
language (str) – Language of the text. Defaults to English (‘en’).
pos_tags (list[str]) – List of POS tags to consider. Defaults to lexical tokens.
threshold – Threshold for the number of synsets. Defaults to 5.

Returns:

Polars DataFrame with the number of high synsets per POS. The columns are named ‘n_high_synsets_{pos}’.

Return type:

data (pl.DataFrame)

elfen.semantic.get_low_synsets_per_pos(data: DataFrame, backbone: str = 'spacy', language: str = 'en', pos_tags: list[str] = ['NOUN', 'VERB', 'ADJ', 'ADV'], threshold: int = 2, **kwargs: dict[str, str]) → DataFrame

Calculates the number of words with a low number of synsets per POS in a text.

Parameters:

data (pl.DataFrame) – Polars DataFrame.
backbone (str) – NLP library used. ‘spacy’ or ‘stanza’.
language (str) – Language of the text. Defaults to English (‘en’).
pos_tags (list[str]) – List of POS tags to consider. Defaults to lexical tokens.
threshold – Threshold for the number of synsets. Defaults to 2.

Returns:

Polars DataFrame with the number of low synsets per POS. The columns are named ‘n_low_synsets_{pos}’.

Return type:

data (pl.DataFrame)

elfen.semantic.get_num_hedges(data: DataFrame, lexicon: list[str], text_column: str = 'text', **kwargs: dict[str, str]) → DataFrame

Calculates the number of hedges in a text.

Parameters:

data (pl.DataFrame) – Polars DataFrame.
lexicon (list[str]) – List of hedges.
text_column (str) – Name of the column containing the text.

Returns:

Polars DataFrame with the number of hedges.

Return type:

data (pl.DataFrame)

elfen.semantic.get_num_high_synsets(data: DataFrame, backbone: str = 'spacy', language: str = 'en', pos_tags: list[str] = ['NOUN', 'VERB', 'ADJ', 'ADV'], threshold: int = 5, **kwargs: dict[str, str]) → DataFrame

Calculates the number of words with a high number of synsets in a text.

Parameters:

data (pl.DataFrame) – Polars DataFrame.
backbone (str) – NLP library used. ‘spacy’ or ‘stanza’.
language (str) – Language of the text. Defaults to English (‘en’).
pos_tags (list[str]) – List of POS tags to consider. Defaults to lexical tokens.
threshold – Threshold for the number of synsets. Defaults to 5.

Returns:

Polars DataFrame with the number of high synsets. The column is named ‘n_high_synsets’.

Return type:

data (pl.DataFrame)

elfen.semantic.get_num_low_synsets(data: DataFrame, backbone: str = 'spacy', language: str = 'en', pos_tags: list[str] = ['NOUN', 'VERB', 'ADJ', 'ADV'], threshold: int = 2, **kwargs: dict[str, str]) → DataFrame

Calculates the number of words with a low number of synsets in a text.

Parameters:

data (pl.DataFrame) – Polars DataFrame.
backbone (str) – NLP library used. ‘spacy’ or ‘stanza’.
language (str) – Language of the text. Defaults to English (‘en’).
pos_tags (list[str]) – List of POS tags to consider. Defaults to lexical tokens.
threshold – Threshold for the number of synsets. Defaults to 2.

Returns:

Polars DataFrame with the number of low synsets. The column is named ‘n_low_synsets’.

Return type:

data (pl.DataFrame)

elfen.semantic.get_synsets(data: DataFrame, backbone: str = 'spacy', language: str = 'en', pos_tags: list[str] = ['NOUN', 'VERB', 'ADJ', 'ADV'], **kwargs: dict[str, str]) → DataFrame

Calculates the number of synsets in a text per token.

WordNet synsets serve as a proxy for the ambiguity/polysemy of a word.

Parameters:

data (pl.DataFrame) – Polars DataFrame.
backbone (str) – NLP library used.
language (str) – Language of the text. Defaults to English (‘en’).
pos_tags (list[str]) – List of POS tags to consider. Defaults to lexical tokens.

Returns:

Polars DataFrame with the numbers of synsets per text. The columns are named ‘synsets’ and ‘synsets_{pos}’.

Return type:

data (pl.DataFrame)

elfen.semantic.load_hedges(hedges_file: str, language: str) → list[str]

Loads the hedges from the given file.

Parameters:: hedges_file (str) – Path to the file containing the hedges.
Returns:: List of hedge words.
Return type:: hedges (list[str])

elfen.surface module

This module contains functions to calculate various surface-level features from text data.

The surface-level features implemented in this module are:

Raw Sequence Length:
The number of characters in the text including whitespaces.
Number of Tokens: The number of tokens in the text.
Number of Sentences: The number of sentences in the text.
Number of Tokens per Sentence: The average number of tokens per sentence.
Number of Characters:
The number of characters in the text excluding whitespaces.
Number of Characters per Sentence:
The average number of characters per sentence.
Raw Length per Sentence:
The average number of characters per sentence including whitespaces.
Average Word Length: The average length of a word in the text.
Number of Types: The number of unique tokens in the text.
Number of Long Words: The number of words longer than a threshold.
Number of Lemmas: The number of unique lemmas in the text.
Token Frequencies: The frequency of each token in the text.

elfen.surface.get_avg_word_length(data: DataFrame, backbone: str = 'spacy', text_column: str = 'text', **kwargs: dict[str, str]) → DataFrame

Calculates the average word length in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the average word length in the text data. The average word length is stored in a new column named ‘avg_word_length’.

Return type:

data (pl.DataFrame)

elfen.surface.get_chars_per_sentence(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the average number of characters per sentence in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the average number of characters per sentence in the text data. The average number of characters per sentence is stored in a new column named ‘characters_per_sentence’.

Return type:

data (pl.DataFrame)

elfen.surface.get_global_lemma_frequencies(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → dict[str, int]

Calculates the global frequency of each lemma in the text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A dictionary containing the frequency of each lemma in the text.

Return type:

lemma_freqs (dict[str, int])

elfen.surface.get_global_token_frequencies(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → dict[str, int]

Calculates the global frequency of each token in the text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A dictionary containing the frequency of each token in the text.

Return type:

token_freqs (dict[str, int])

elfen.surface.get_num_characters(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of characters in a text. Only takes tokens into account in contrast to get_raw_sequence_length.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of characters in the text data. The number of characters is stored in a new column named ‘n_characters’.

Return type:

data (pl.DataFrame)

elfen.surface.get_num_lemmas(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of unique lemmas in the text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of unique lemmas in the text data. The number of unique lemmas is stored in a new column named ‘n_lemmas’.

Return type:

data (pl.DataFrame)

elfen.surface.get_num_long_words(data: DataFrame, backbone: str = 'spacy', threshold: int = 6, **kwargs: dict[str, str]) → DataFrame

Calculates the number of long words in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.
threshold (int) – The minimum length of a word to be considered long.

Returns:

A Polars DataFrame containing the number of long words in the text data. The number of long words is stored in a new column named ‘n_long_words’.

Return type:

data (pl.DataFrame)

elfen.surface.get_num_sentences(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of sentences in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of sentences in the text data. The number of sentences is stored in a new column named ‘n_sentences’.

Return type:

data (pl.DataFrame)

elfen.surface.get_num_tokens(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the sequence length (number of tokens) of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the sequence length of the text data. The sequence length is stored in a new column named ‘n_tokens’.

Return type:

data (pl.DataFrame)

elfen.surface.get_num_tokens_per_sentence(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the average number of tokens per sentence in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the average number of tokens per sentence in the text data. The average number of tokens per sentence is stored in a new column named ‘tokens_per_sentence’.

Return type:

data (pl.DataFrame)

elfen.surface.get_num_types(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the number of types in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the number of types in the text data. The number of types is stored in a new column named ‘n_types’.

Return type:

data (pl.DataFrame)

elfen.surface.get_raw_length_per_sentence(data: DataFrame, backbone: str = 'spacy', text_column: str = 'text', **kwargs: dict[str, str]) → DataFrame

Calculates the average number of characters per sentence in a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the average number of characters per sentence in the text data. The average number of characters per sentence is stored in a new column named ‘raw_length_per_sentence’.

Return type:

data (pl.DataFrame)

elfen.surface.get_raw_sequence_length(data: DataFrame, text_column: str = 'text', **kwargs: dict[str, str]) → DataFrame

Calculates the raw text length (number of characters) of a text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
text_column (str) – The name of the column containing the text data.

Returns:

A Polars DataFrame containing the raw text length of the text data. The raw text length is stored in a new column named ‘raw_sequence_length’.

Return type:

data (pl.DataFrame)

elfen.surface.get_token_freqs(data: DataFrame, backbone: str = 'spacy', **kwargs: dict[str, str]) → DataFrame

Calculates the frequency of each token in the text.

Parameters:

data (pl.DataFrame) – A Polars DataFrame containing the text data.
backbone (str) – The NLP library used to process the text data. Either ‘spacy’ or ‘stanza’.

Returns:

A Polars DataFrame containing the frequency of each token in the text data. The frequency of each token is stored in a new column named ‘token_freqs’.

Return type:

data (pl.DataFrame)

elfen.util module

This module contains utility functions for working with Polars DataFrames.

elfen.util.normalize_column(data: DataFrame, column: str) → DataFrame

Normalizes a column to have a mean of 0 and a standard deviation of 1.

Parameters:

data (pl.DataFrame) – A Polars DataFrame.
column (str) – The name of the column to normalize.

Returns:

A Polars DataFrame with the column normalized

Return type:

normalized_data (pl.DataFrame)

elfen.util.rescale_column(data: DataFrame, column: str, minimum: float = 0.0, maximum: float = 1.0) → DataFrame

Rescales a column to a custom range. Defaults to [0, 1].

Parameters:

data (pl.DataFrame) – A Polars DataFrame.
column (str) – The name of the column to rescale.
minimum (float) – The desired minimum value of the column. Defaults to 0.
maximum (float) – The desired maximum value of the column. Defaults to 1.

Returns:

A Polars DataFrame with the column rescaled to [minimum, maximum].

Return type:

rescaled_data (pl.DataFrame)

elfen.util.upos_to_wn(upos_tag: str) → str

Converts a Universal POS tag to a (Senti)WordNet POS tag.

Parameters:: upos_tag (str) – A Universal POS tag.
Returns:: A WordNet POS tag.
Return type:: wn_tag (str)

elfen.util.zero_token_warning_nan(feature: str) → None

Generates a warning message for features that cannot be calculated for texts with zero tokens. Warning is to be issued if any NaN values are present in the feature.

Parameters:: feature (str) – The name of the feature.
Returns:: None

elfen.util.zero_token_warning_null(feature: str) → None

Generates a warning message for features that cannot be calculated for texts with zero tokens. Warning is to be issued if any Null values are present in the feature.

Parameters:: feature (str) – The name of the feature.
Returns:: None