From 5fb3551fb7a4f6eb6d1d13d5e5d2c35a32d7616d Mon Sep 17 00:00:00 2001 From: Henri Froese Date: Tue, 4 Aug 2020 18:49:28 +0200 Subject: [PATCH 01/13] First draft of Tutorial on the different series types Focus on Representation Series Co-authored-by: Maximilian Krahn --- .../docs/different_series_types_tutorial.md | 86 +++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 website/docs/different_series_types_tutorial.md diff --git a/website/docs/different_series_types_tutorial.md b/website/docs/different_series_types_tutorial.md new file mode 100644 index 00000000..1be46d66 --- /dev/null +++ b/website/docs/different_series_types_tutorial.md @@ -0,0 +1,86 @@ +

Pandas Series Types in Texthero

+ +In Texthero, we're always working with Pandas Series and Pandas Dataframes to hold a (possibly very large) collection of documents. To make things easier and more intuitive, we differentiate between 4 different types of Series, depending on the cell content. For example, the functions in preprocessing.py usually take as input a Series where every cell is a string, and return as output a Series where every cell is a string. + +These are the implemented types: + +1. TextSeries: Every cell is a text, i.e. a string. For example, +`pd.Series(["test", "test"])` is a valid TextSeries. + +2. TokenSeries: Every cell is a list of words/tokens, i.e. a list +of strings. For example, `pd.Series([["test"], ["token2", "token3"]])` is a valid TokenSeries. + +3. VectorSeries: Every cell is a vector representing text, i.e. +a list of floats. For example, `pd.Series([[1.0, 2.0], [3.0]])` is a valid VectorSeries. + +4. RepresentationSeries: Series is multiindexed with level one +being the document, level two being the individual features and their values. +For example, +`pd.Series([1, 2, 3], index=pd.MultiIndex.from_tuples([("doc1", "word1"), ("doc1", "word2"), ("doc2", "word1")]))` +is a valid RepresentationSeries. + +Now, if you see a function in the documentation that looks like this: +``` +def tfidf(s: TokenSeries) -> RepresentationSeries +``` + +then you know that the function takes a Pandas Series +whose cells are lists of strings (tokens) and will +return a Pandas Series whose cells are lists of floats. And this function: +``` +def pca(s: Union[VectorSeries, RepresentationSeries) -> VectorSeries +``` +can handle both _VectorSeries_ and _RepresentationSeries_ as input and always returns a _VectorSeries_. + + + +

Representation Series

+ +As you can see, the `RepresentationSeries` type is a little more complex than the others. Let's have a closer look to see what it is and why, where and how it is used! + +

What is it?

+ +A _RepresentationSeries_ is multiindexed with level one +being the document, and level two being the individual features and their values. It could look like this: + +```python +>>> import texthero as hero +>>> import pandas as pd +>>> s = pd.Series(["Sentence one one", "Sentence two"]) +>>> s.pipe(hero.tokenize).pipe(hero.count) # first tokenize Series, then calculate word count +document word +0 Sentence 1 + one 2 +1 Sentence 1 + two 1 +dtype: Sparse[int64, 0] +``` + +The output shown is a _RepresentationSeries_! It just means that we have a level for each document, and in that level we can see the individual features of the document. + + +

Why is it used?

+ +You might have noticed the `dtype: Sparse[int64, 0]` in the last code bit. That's precisely the reason we use this: Pandas internally does not store the zeros we get when e.g. calculating `hero.count`; so we don't see that the word "two" has zero occurrences in the first sentence, it's not stored. This is a massive advantage when dealing with *big data*: In a _RepresentationSeries_, we only store the data that's relevant for each document to save time and space! + +

When and how is it used? Do I have to work with multiindexes?!

+ +The _RepresentationSeries_ is mostly used internally for performance reasons. For example, as you can see above, the default output from `hero.count` is such a Series, but if you apply e.g. `hero.pca` afterwards, you don't even notice the complex _RepresentationSeries_: `s.pipe(hero.count).pipe(hero.pca)` works just fine; everything is seamlessly integrated in the library. + +The only thing you cannot do is store a _RepresentationSeries_ in your dataframe, as the indexes are different. If you really want to do this, you can use `hero.flatten`: + +```python +>>> import texthero as hero +>>> import pandas as pd +>>> s = pd.Series(["Sentence one one", "Sentence two"]) +>>> df = pd.DataFrame(s) +>>> df["count"] = s.pipe(hero.tokenize).pipe(hero.count) # WRONG +>>> # ERROR: cannot put RepresentationSeries into the DataFrame +>>> # INSTEAD DO THIS: +>>> df["count"] = s.pipe(hero.tokenize).pipe(hero.count).pipe(hero.flatten) +>>> df + 0 count +0 Sentence one one [1, 2.0, 0.0] +1 Sentence two [1, 0.0, 1.0] +``` +As you can see, we then lose the advantage of _sparseness_ (i.e. not storing the zeroes): The third word, "two", is now also stored for the first sentence with "0.0" occurrences. From 9f76e274f7fe7154291c800cdb01bc517ea90c3c Mon Sep 17 00:00:00 2001 From: Henri Froese Date: Fri, 7 Aug 2020 20:31:15 +0200 Subject: [PATCH 02/13] Improve HeroSeries tutorial - rename to getting-started-heroseries - adapt suggested structure - improve content --- .../docs/different_series_types_tutorial.md | 86 ---------- website/docs/getting-started-heroseries.md | 155 ++++++++++++++++++ 2 files changed, 155 insertions(+), 86 deletions(-) delete mode 100644 website/docs/different_series_types_tutorial.md create mode 100644 website/docs/getting-started-heroseries.md diff --git a/website/docs/different_series_types_tutorial.md b/website/docs/different_series_types_tutorial.md deleted file mode 100644 index 1be46d66..00000000 --- a/website/docs/different_series_types_tutorial.md +++ /dev/null @@ -1,86 +0,0 @@ -

Pandas Series Types in Texthero

- -In Texthero, we're always working with Pandas Series and Pandas Dataframes to hold a (possibly very large) collection of documents. To make things easier and more intuitive, we differentiate between 4 different types of Series, depending on the cell content. For example, the functions in preprocessing.py usually take as input a Series where every cell is a string, and return as output a Series where every cell is a string. - -These are the implemented types: - -1. TextSeries: Every cell is a text, i.e. a string. For example, -`pd.Series(["test", "test"])` is a valid TextSeries. - -2. TokenSeries: Every cell is a list of words/tokens, i.e. a list -of strings. For example, `pd.Series([["test"], ["token2", "token3"]])` is a valid TokenSeries. - -3. VectorSeries: Every cell is a vector representing text, i.e. -a list of floats. For example, `pd.Series([[1.0, 2.0], [3.0]])` is a valid VectorSeries. - -4. RepresentationSeries: Series is multiindexed with level one -being the document, level two being the individual features and their values. -For example, -`pd.Series([1, 2, 3], index=pd.MultiIndex.from_tuples([("doc1", "word1"), ("doc1", "word2"), ("doc2", "word1")]))` -is a valid RepresentationSeries. - -Now, if you see a function in the documentation that looks like this: -``` -def tfidf(s: TokenSeries) -> RepresentationSeries -``` - -then you know that the function takes a Pandas Series -whose cells are lists of strings (tokens) and will -return a Pandas Series whose cells are lists of floats. And this function: -``` -def pca(s: Union[VectorSeries, RepresentationSeries) -> VectorSeries -``` -can handle both _VectorSeries_ and _RepresentationSeries_ as input and always returns a _VectorSeries_. - - - -

Representation Series

- -As you can see, the `RepresentationSeries` type is a little more complex than the others. Let's have a closer look to see what it is and why, where and how it is used! - -

What is it?

- -A _RepresentationSeries_ is multiindexed with level one -being the document, and level two being the individual features and their values. It could look like this: - -```python ->>> import texthero as hero ->>> import pandas as pd ->>> s = pd.Series(["Sentence one one", "Sentence two"]) ->>> s.pipe(hero.tokenize).pipe(hero.count) # first tokenize Series, then calculate word count -document word -0 Sentence 1 - one 2 -1 Sentence 1 - two 1 -dtype: Sparse[int64, 0] -``` - -The output shown is a _RepresentationSeries_! It just means that we have a level for each document, and in that level we can see the individual features of the document. - - -

Why is it used?

- -You might have noticed the `dtype: Sparse[int64, 0]` in the last code bit. That's precisely the reason we use this: Pandas internally does not store the zeros we get when e.g. calculating `hero.count`; so we don't see that the word "two" has zero occurrences in the first sentence, it's not stored. This is a massive advantage when dealing with *big data*: In a _RepresentationSeries_, we only store the data that's relevant for each document to save time and space! - -

When and how is it used? Do I have to work with multiindexes?!

- -The _RepresentationSeries_ is mostly used internally for performance reasons. For example, as you can see above, the default output from `hero.count` is such a Series, but if you apply e.g. `hero.pca` afterwards, you don't even notice the complex _RepresentationSeries_: `s.pipe(hero.count).pipe(hero.pca)` works just fine; everything is seamlessly integrated in the library. - -The only thing you cannot do is store a _RepresentationSeries_ in your dataframe, as the indexes are different. If you really want to do this, you can use `hero.flatten`: - -```python ->>> import texthero as hero ->>> import pandas as pd ->>> s = pd.Series(["Sentence one one", "Sentence two"]) ->>> df = pd.DataFrame(s) ->>> df["count"] = s.pipe(hero.tokenize).pipe(hero.count) # WRONG ->>> # ERROR: cannot put RepresentationSeries into the DataFrame ->>> # INSTEAD DO THIS: ->>> df["count"] = s.pipe(hero.tokenize).pipe(hero.count).pipe(hero.flatten) ->>> df - 0 count -0 Sentence one one [1, 2.0, 0.0] -1 Sentence two [1, 0.0, 1.0] -``` -As you can see, we then lose the advantage of _sparseness_ (i.e. not storing the zeroes): The third word, "two", is now also stored for the first sentence with "0.0" occurrences. diff --git a/website/docs/getting-started-heroseries.md b/website/docs/getting-started-heroseries.md new file mode 100644 index 00000000..6cfd801c --- /dev/null +++ b/website/docs/getting-started-heroseries.md @@ -0,0 +1,155 @@ +

HeroSeries

+ +In Texthero, we're always working with Pandas Series and Pandas Dataframes to gain insights from text data! To make things easier and more intuitive, we differentiate between different types of Series, depending on where we are on the road to understanding our dataset. + +

Overview

+ +When working with text data, it is easy to get overwhelmed by the many different functions that can be applied to the data. We want to make the whole journey as clear as possible. For example, when we start working with a new dataset, we usually want to do some preprocessing first. At the beginning, the data is in a DataFrame or Series where every document is one string. It might look like this: +```python + text +document_id +0 "Text in the first document" +1 "Text in the second document" +2 "Text in the third document" +3 "Text in the fourth document" +4 ... + +``` + + Consequently, in the Texthero's _preprocessing_ module, the functions usually take as input a Series where every cell is a string, and return as output a Series where every cell is a string. We will call these kinds of Series _TextSeries_, so users know immediately what kind of Series the functions can work on. For example, you might see a function + ```python +remove_punctuation(s: TextSeries) -> TextSeries + ``` +in the documentation. You then know that this can be used on a DataFrame or Series in the preprocessing phase of your work, where each document is one string. + +

The four HeroSeries Types

+ +These are the four types currently supported by the library; almost all of the libraries functions takes as input and return as output one of these types: + +1. **TextSeries**: Every cell is a text, i.e. a string. For example, +`pd.Series(["test", "test"])` is a valid TextSeries. + +2. **TokenSeries**: Every cell is a list of words/tokens, i.e. a list +of strings. For example, `pd.Series([["test"], ["token2", "token3"]])` is a valid TokenSeries. + +3. **VectorSeries**: Every cell is a vector representing text, i.e. +a list of floats. For example, `pd.Series([[1.0, 2.0], [3.0]])` is a valid VectorSeries. + +4. **RepresentationSeries**: Series is multiindexed with level one +being the document, level two being the individual features and their values. +For example, +`pd.Series([1, 2, 3], index=pd.MultiIndex.from_tuples([("doc1", "word1"), ("doc1", "word2"), ("doc2", "word1")]))` +is a valid RepresentationSeries. + +Now, if you see a function in the documentation that looks like this: +```python +tfidf(s: TokenSeries) -> RepresentationSeries +``` + +then you know that the function takes a Pandas Series +whose cells are lists of strings (tokens) and will +return a Pandas Series whose cells are lists of floats. +You might call it like this: +```python +>>> import texthero as hero +>>> import pandas as pd +>>> s = pd.Series(["Text of first document", "Text of second document"]) +>>> s_tfidf = s.pipe(hero.tokenize).pipe(hero.tfidf) +``` + + +And this function: +```python +pca(s: RepresentationSeries) -> VectorSeries +``` +needs a _RepresentationSeries_ as input and always returns a _VectorSeries_. + +

The Types in Detail

+ +We'll now have a closer look at each of the types and learn where they are used in a typical NLP workflow. + +

TextSeries

+ +In a _TextSeries_, every cell is a string. As we saw at the beginning of this tutorial, this type is mostly used in preprocessing. It is very simple and allows us to easily clean a text dataset. Additionally, many NLP functions such as `named_entities, noun_chunks, pos_tag` take a _TextSeries_ as input. + +Example of a function that takes and returns a _TextSeries_: +```python +>>> s = pd.Series(["Text: of first! document", "Text of second ... document"]) +>>> hero.remove_punctuation(s) +0 text first document +1 text second document +dtype: object +``` + +

TokenSeries

+ +In a _TokenSeries_, every cell is a list of words/tokens. We use this to prepare our data for _representation_, so to gain insights from it through mathematical methods. This is why the functions that initially transform your documents to vectors, namely `tfidf, term_frequency, count`, take a _TokenSeries_ as input. + +Example of a function that takes a _TextSeries_ and returns a _TokenSeries_: +```python +>>> s = pd.Series(["text first document", "text second document"]) +>>> hero.tokenize(s) +0 [text, first, document] +1 [text, second, document] +dtype: object +``` + +

VectorSeries

+ +In a _VectorSeries_, every cell is a vector representing text. We use this when we have a low-dimensional (e.g. lists with length 2 or 3), dense (so not a lot of zeroes) representation of our texts that we want to work on. For example, the dimensionality reduction functions `pca, nmf, tsne` all take a high-dimensional representation of our text in the form of a _RepresentationSeries_ (see below), and return a low-dimensional representation of our text in the form of a _VectorSeries_. + +Example of a function that takes as input a _RepresentationSeries_ and returns a _VectorSeries_: +```python +>>> s = pd.Series(["text first document", "text second document"]).pipe(hero.tokenize).pipe(hero.term_frequency) +>>> hero.pca(s) +0 [0.118, 0.0] +1 [-0.118, 0.0] +dtype: object +``` + +

Representation Series

+ +As you can see, the `RepresentationSeries` type is a little more complex than the others. Let's have a closer look to see what it is and why, where and how it is used! + +

What is it?

+ +A _RepresentationSeries_ is multiindexed with level one +being the document, and level two being the individual features and their values. It could look like this: + +```python +>>> s = pd.Series(["Sentence one one", "Sentence two"]) +>>> s.pipe(hero.tokenize).pipe(hero.count) # first tokenize Series, then calculate word count +document word +0 Sentence 1 + one 2 +1 Sentence 1 + two 1 +dtype: Sparse[int64, 0] +``` + +The output shown is a _RepresentationSeries_! It just means that we have a level for each document, and in that level we can see the individual features of the document. + + +

Why is it used?

+ +You might have noticed the `dtype: Sparse[int64, 0]` in the last code bit. That's precisely the reason we use this: Pandas internally does not store the zeros we get when e.g. calculating `hero.count`; so we don't see that the word "two" has zero occurrences in the first sentence, it's not stored. This is a massive advantage when dealing with *big data*: In a _RepresentationSeries_, we only store the data that's relevant for each document to save time and space! + +

When and how is it used? Do I have to work with multiindexes?!

+ +The _RepresentationSeries_ is mostly used internally for performance reasons. For example, as you can see above, the default output from `hero.count` is such a Series, but if you apply e.g. `hero.pca` afterwards, you don't even notice the complex _RepresentationSeries_: `s.pipe(hero.count).pipe(hero.pca)` works just fine; everything is seamlessly integrated in the library. + +The only thing you cannot do is store a _RepresentationSeries_ in your dataframe, as the indexes are different. If you really want to do this, you can use `hero.flatten`: + +```python +>>> s = pd.Series(["Sentence one one", "Sentence two"]) +>>> df = pd.DataFrame(s) +>>> df["count"] = s.pipe(hero.tokenize).pipe(hero.count) # WRONG +>>> # ERROR: cannot put RepresentationSeries into the DataFrame +>>> # INSTEAD DO THIS: +>>> df["count"] = s.pipe(hero.tokenize).pipe(hero.count).pipe(hero.flatten) +>>> df + 0 count +0 Sentence one one [1, 2.0, 0.0] +1 Sentence two [1, 0.0, 1.0] +``` +As you can see, we then lose the advantage of _sparseness_ (i.e. not storing the zeroes): The third word, "two", is now also stored for the first sentence with "0.0" occurrences. This is why we strongly suggest you avoid using `hero.flatten` at all; Texthero is designed with performance in mind and using `hero.flatten` goes against that goal! From df8b3142d637211f79535f2ab72661acd77b22f9 Mon Sep 17 00:00:00 2001 From: Henri Froese Date: Sat, 22 Aug 2020 14:15:23 +0200 Subject: [PATCH 03/13] Incorporate change RepresentationSeries -> DocumentTermDF Co-authored-by: Maximilian Krahn --- website/docs/getting-started-heroseries.md | 91 ++++++++++++---------- 1 file changed, 50 insertions(+), 41 deletions(-) diff --git a/website/docs/getting-started-heroseries.md b/website/docs/getting-started-heroseries.md index 6cfd801c..4bc7415d 100644 --- a/website/docs/getting-started-heroseries.md +++ b/website/docs/getting-started-heroseries.md @@ -1,6 +1,6 @@

HeroSeries

-In Texthero, we're always working with Pandas Series and Pandas Dataframes to gain insights from text data! To make things easier and more intuitive, we differentiate between different types of Series, depending on where we are on the road to understanding our dataset. +In Texthero, we're always working with Pandas Series and Pandas Dataframes to gain insights from text data! To make things easier and more intuitive, we differentiate between different types of Series/DataFrames, depending on where we are on the road to understanding our dataset.

Overview

@@ -16,7 +16,7 @@ document_id ``` - Consequently, in the Texthero's _preprocessing_ module, the functions usually take as input a Series where every cell is a string, and return as output a Series where every cell is a string. We will call these kinds of Series _TextSeries_, so users know immediately what kind of Series the functions can work on. For example, you might see a function + Consequently, in the Texthero's _preprocessing_ module, the functions usually take as input a Series where every cell is a string, and return as output a Series where every cell is a string. We will call this kind of Series _TextSeries_, so users know immediately what kind of Series the functions can work on. For example, you might see a function ```python remove_punctuation(s: TextSeries) -> TextSeries ``` @@ -35,20 +35,20 @@ of strings. For example, `pd.Series([["test"], ["token2", "token3"]])` is a vali 3. **VectorSeries**: Every cell is a vector representing text, i.e. a list of floats. For example, `pd.Series([[1.0, 2.0], [3.0]])` is a valid VectorSeries. -4. **RepresentationSeries**: Series is multiindexed with level one -being the document, level two being the individual features and their values. +4. **DocumentTermDF**: A DataFrame where the rows are the documents and the columns are the words/terms in all the documents. The columns are multiindexed with level one +being the content name (e.g. "tfidf"), level two being the individual features and their values. For example, -`pd.Series([1, 2, 3], index=pd.MultiIndex.from_tuples([("doc1", "word1"), ("doc1", "word2"), ("doc2", "word1")]))` +`pd.DataFrame([[1, 2, 3], [4,5,6]], columns=pd.MultiIndex.from_tuples([("count", "hi"), ("count", "servus"), ("count", "hola")]))` is a valid RepresentationSeries. Now, if you see a function in the documentation that looks like this: ```python -tfidf(s: TokenSeries) -> RepresentationSeries +tfidf(s: TokenSeries) -> DocumentTermDF ``` then you know that the function takes a Pandas Series whose cells are lists of strings (tokens) and will -return a Pandas Series whose cells are lists of floats. +return a Pandas DataFrame with the individual features. You might call it like this: ```python >>> import texthero as hero @@ -60,9 +60,9 @@ You might call it like this: And this function: ```python -pca(s: RepresentationSeries) -> VectorSeries +pca(s: Union[VectorSeries, DocumentTermDF]) -> VectorSeries ``` -needs a _RepresentationSeries_ as input and always returns a _VectorSeries_. +needs a _DocumentTermDF_ or _VectorSeries_ as input and always returns a _VectorSeries_.

The Types in Detail

@@ -75,7 +75,7 @@ In a _TextSeries_, every cell is a string. As we saw at the beginning of this tu Example of a function that takes and returns a _TextSeries_: ```python >>> s = pd.Series(["Text: of first! document", "Text of second ... document"]) ->>> hero.remove_punctuation(s) +>>> hero.clean(s) 0 text first document 1 text second document dtype: object @@ -96,9 +96,9 @@ dtype: object

VectorSeries

-In a _VectorSeries_, every cell is a vector representing text. We use this when we have a low-dimensional (e.g. lists with length 2 or 3), dense (so not a lot of zeroes) representation of our texts that we want to work on. For example, the dimensionality reduction functions `pca, nmf, tsne` all take a high-dimensional representation of our text in the form of a _RepresentationSeries_ (see below), and return a low-dimensional representation of our text in the form of a _VectorSeries_. +In a _VectorSeries_, every cell is a vector representing text. We use this when we have a low-dimensional (e.g. vectors with length <=1000), dense (so not a lot of zeroes) representation of our texts that we want to work on. For example, the dimensionality reduction functions `pca, nmf, tsne` all take a high-dimensional representation of our text (in the form of a _DocumentTermDF_ (see below) or _VectorSeries_, and return a low-dimensional representation of our text in the form of a _VectorSeries_. -Example of a function that takes as input a _RepresentationSeries_ and returns a _VectorSeries_: +Example of a function that takes as input a _DocumentTermDF_ or _VectorSeries_ and returns a _VectorSeries_: ```python >>> s = pd.Series(["text first document", "text second document"]).pipe(hero.tokenize).pipe(hero.term_frequency) >>> hero.pca(s) @@ -107,49 +107,58 @@ Example of a function that takes as input a _RepresentationSeries_ and returns a dtype: object ``` -

Representation Series

+

DocumentTermDF

-As you can see, the `RepresentationSeries` type is a little more complex than the others. Let's have a closer look to see what it is and why, where and how it is used! +As you can see, the `DocumentTermDF` type is a little more complex than the others. Let's have a closer look to see what it is and why, where and how it is used!

What is it?

-A _RepresentationSeries_ is multiindexed with level one -being the document, and level two being the individual features and their values. It could look like this: +A _DocumentTermDF_ is a Pandas implementation of a [Document Term Matrix](https://en.wikipedia.org/wiki/Document-term_matrix), so the rows are the documents and the columns are the words/terms in all the documents. A _DocumentTermDF_ has multiindexed columns with level one +being the content name (e.g. "tfidf"), and level two being the terms. It could look like this: ```python >>> s = pd.Series(["Sentence one one", "Sentence two"]) ->>> s.pipe(hero.tokenize).pipe(hero.count) # first tokenize Series, then calculate word count -document word -0 Sentence 1 - one 2 -1 Sentence 1 - two 1 -dtype: Sparse[int64, 0] +>>> t = s.pipe(hero.tokenize).pipe(hero.count) # first tokenize Series, then calculate word count +>>> t + count + Sentence one two +0 1 2 0 +1 1 0 1 ``` -The output shown is a _RepresentationSeries_! It just means that we have a level for each document, and in that level we can see the individual features of the document. -

Why is it used?

-You might have noticed the `dtype: Sparse[int64, 0]` in the last code bit. That's precisely the reason we use this: Pandas internally does not store the zeros we get when e.g. calculating `hero.count`; so we don't see that the word "two" has zero occurrences in the first sentence, it's not stored. This is a massive advantage when dealing with *big data*: In a _RepresentationSeries_, we only store the data that's relevant for each document to save time and space! +Have a look at the datatypes of `t` from the last code bit: +```python +>>> t.dtypes +count Sentence Sparse[int64, 0] + one Sparse[int64, 0] + two Sparse[int64, 0] +``` +That's precisely the reason we use this: Pandas internally does not store the zeros we get when e.g. calculating `hero.count`; so we don't see that the word "two" has zero occurrences in the first sentence, it's not stored. This is a massive advantage when dealing with *big data*: In a _DocumentTermDF_, we only store the data that's relevant for each document to save lots and lots of time and space! + +Let's look at an example with some more data. +```python +>>> data = pd.read_csv("https://github.com/jbesomi/texthero/raw/master/dataset/bbcsport.csv") +>>> data_count = data["text"].pipe(count) +>>> data_count.sparse.density +0.012... +``` +We can see that only around 1.2% of our _DocumentTermDF_ `data_count` is filled, so using the sparse DataFrame is saving us a lot of space.

When and how is it used? Do I have to work with multiindexes?!

-The _RepresentationSeries_ is mostly used internally for performance reasons. For example, as you can see above, the default output from `hero.count` is such a Series, but if you apply e.g. `hero.pca` afterwards, you don't even notice the complex _RepresentationSeries_: `s.pipe(hero.count).pipe(hero.pca)` works just fine; everything is seamlessly integrated in the library. - -The only thing you cannot do is store a _RepresentationSeries_ in your dataframe, as the indexes are different. If you really want to do this, you can use `hero.flatten`: +The _DocumentTermDF_ is mostly used internally for performance reasons. For example, as you can see above, the default output from `hero.count` is such a Series, but if you apply e.g. `hero.pca` afterwards, you don't even notice the _DocumentTermDF_: `s.pipe(hero.count).pipe(hero.pca)` works just fine; everything is seamlessly integrated in the library. +The only thing you _can_ but _should not_ do is store a _DocumentTermDF_ in your dataframe, as the performance is really bad. If you really want to, here's the two options: ```python ->>> s = pd.Series(["Sentence one one", "Sentence two"]) ->>> df = pd.DataFrame(s) ->>> df["count"] = s.pipe(hero.tokenize).pipe(hero.count) # WRONG ->>> # ERROR: cannot put RepresentationSeries into the DataFrame ->>> # INSTEAD DO THIS: ->>> df["count"] = s.pipe(hero.tokenize).pipe(hero.count).pipe(hero.flatten) ->>> df - 0 count -0 Sentence one one [1, 2.0, 0.0] -1 Sentence two [1, 0.0, 1.0] -``` -As you can see, we then lose the advantage of _sparseness_ (i.e. not storing the zeroes): The third word, "two", is now also stored for the first sentence with "0.0" occurrences. This is why we strongly suggest you avoid using `hero.flatten` at all; Texthero is designed with performance in mind and using `hero.flatten` goes against that goal! +>>> data = pd.read_csv("https://github.com/jbesomi/texthero/raw/master/dataset/bbcsport.csv") +>>> data_count = data["text"].pipe(count) + +>>> # Option 1: recommended if you really want to put the DocumenTermDF into your DataFrame +>>> data = pd.concat(data, data_count) + +>>> # Option 1: not recommended as performance is not optimal +>>> data["count"] = data_count +``` \ No newline at end of file From 7fe8c5446026c6bc9f141466a294db039bd2890e Mon Sep 17 00:00:00 2001 From: Henri Froese Date: Sat, 22 Aug 2020 14:52:23 +0200 Subject: [PATCH 04/13] Incorporate suggested changes --- ...ing-started-heroseries.md => getting-started-herotypes.md} | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) rename website/docs/{getting-started-heroseries.md => getting-started-herotypes.md} (98%) diff --git a/website/docs/getting-started-heroseries.md b/website/docs/getting-started-herotypes.md similarity index 98% rename from website/docs/getting-started-heroseries.md rename to website/docs/getting-started-herotypes.md index 4bc7415d..f59465e5 100644 --- a/website/docs/getting-started-heroseries.md +++ b/website/docs/getting-started-herotypes.md @@ -1,4 +1,4 @@ -

HeroSeries

+

HeroTypes

In Texthero, we're always working with Pandas Series and Pandas Dataframes to gain insights from text data! To make things easier and more intuitive, we differentiate between different types of Series/DataFrames, depending on where we are on the road to understanding our dataset. @@ -149,7 +149,7 @@ We can see that only around 1.2% of our _DocumentTermDF_ `data_count` is filled,

When and how is it used? Do I have to work with multiindexes?!

-The _DocumentTermDF_ is mostly used internally for performance reasons. For example, as you can see above, the default output from `hero.count` is such a Series, but if you apply e.g. `hero.pca` afterwards, you don't even notice the _DocumentTermDF_: `s.pipe(hero.count).pipe(hero.pca)` works just fine; everything is seamlessly integrated in the library. +The _DocumentTermDF_ is mostly used internally for performance reasons. For example, as you can see above, the default output from `hero.count` is such a Series, but if you apply e.g. `hero.pca` afterwards, you don't even notice the _DocumentTermDF_: `s.pipe(hero.count).pipe(hero.normalize).pipe(hero.pca)` works just fine; everything is seamlessly integrated in the library. The only thing you _can_ but _should not_ do is store a _DocumentTermDF_ in your dataframe, as the performance is really bad. If you really want to, here's the two options: ```python From 6de6f35b8bf4c9228c61287291163591139d2d10 Mon Sep 17 00:00:00 2001 From: Henri Froese Date: Sat, 22 Aug 2020 17:14:14 +0200 Subject: [PATCH 05/13] small fixes --- website/docs/getting-started-herotypes.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/website/docs/getting-started-herotypes.md b/website/docs/getting-started-herotypes.md index f59465e5..c620ce2e 100644 --- a/website/docs/getting-started-herotypes.md +++ b/website/docs/getting-started-herotypes.md @@ -149,16 +149,13 @@ We can see that only around 1.2% of our _DocumentTermDF_ `data_count` is filled,

When and how is it used? Do I have to work with multiindexes?!

-The _DocumentTermDF_ is mostly used internally for performance reasons. For example, as you can see above, the default output from `hero.count` is such a Series, but if you apply e.g. `hero.pca` afterwards, you don't even notice the _DocumentTermDF_: `s.pipe(hero.count).pipe(hero.normalize).pipe(hero.pca)` works just fine; everything is seamlessly integrated in the library. +The _DocumentTermDF_ is mostly used internally for performance reasons. For example, as you can see above, the default output from `hero.count` is such a DataFrame, but if you apply e.g. `hero.pca` afterwards, you don't even notice the _DocumentTermDF_: `s.pipe(hero.count).pipe(hero.normalize).pipe(hero.pca)` works just fine; everything is seamlessly integrated in the library. -The only thing you _can_ but _should not_ do is store a _DocumentTermDF_ in your dataframe, as the performance is really bad. If you really want to, here's the two options: +The only thing you _can_ but _should not_ do is store a _DocumentTermDF_ in your dataframe, as the performance is really bad. If you really want to, you can do it like this like this: ```python >>> data = pd.read_csv("https://github.com/jbesomi/texthero/raw/master/dataset/bbcsport.csv") >>> data_count = data["text"].pipe(count) ->>> # Option 1: recommended if you really want to put the DocumenTermDF into your DataFrame ->>> data = pd.concat(data, data_count) - ->>> # Option 1: not recommended as performance is not optimal +>>> # NOT recommended as performance is not optimal >>> data["count"] = data_count ``` \ No newline at end of file From a41ca8b87553fc28636e6c3632a56aaab1ab214c Mon Sep 17 00:00:00 2001 From: Maximilian Krahn Date: Sat, 5 Sep 2020 18:56:37 +0200 Subject: [PATCH 06/13] updated getting started hero types --- website/docs/getting-started-herotypes.md | 35 +++++++++++------------ 1 file changed, 16 insertions(+), 19 deletions(-) diff --git a/website/docs/getting-started-herotypes.md b/website/docs/getting-started-herotypes.md index c620ce2e..0215d632 100644 --- a/website/docs/getting-started-herotypes.md +++ b/website/docs/getting-started-herotypes.md @@ -35,15 +35,14 @@ of strings. For example, `pd.Series([["test"], ["token2", "token3"]])` is a vali 3. **VectorSeries**: Every cell is a vector representing text, i.e. a list of floats. For example, `pd.Series([[1.0, 2.0], [3.0]])` is a valid VectorSeries. -4. **DocumentTermDF**: A DataFrame where the rows are the documents and the columns are the words/terms in all the documents. The columns are multiindexed with level one -being the content name (e.g. "tfidf"), level two being the individual features and their values. +4. **DataFrame**: A Pandas DataFrame where the rows can be the documents and the columns can be the words/terms in all the documents. For example, -`pd.DataFrame([[1, 2, 3], [4,5,6]], columns=pd.MultiIndex.from_tuples([("count", "hi"), ("count", "servus"), ("count", "hola")]))` -is a valid RepresentationSeries. +`pd.DataFrame([[1, 2, 3], [4,5,6]], columns=["hi", "servus", "hola"])` +is a valid DataFrame. Now, if you see a function in the documentation that looks like this: ```python -tfidf(s: TokenSeries) -> DocumentTermDF +tfidf(s: TokenSeries) -> DataFrame ``` then you know that the function takes a Pandas Series @@ -60,9 +59,9 @@ You might call it like this: And this function: ```python -pca(s: Union[VectorSeries, DocumentTermDF]) -> VectorSeries +pca(s: Union[VectorSeries, DataFrame]) -> VectorSeries ``` -needs a _DocumentTermDF_ or _VectorSeries_ as input and always returns a _VectorSeries_. +needs a _DataFrame_ or _VectorSeries_ as input and always returns a _VectorSeries_.

The Types in Detail

@@ -96,9 +95,9 @@ dtype: object

VectorSeries

-In a _VectorSeries_, every cell is a vector representing text. We use this when we have a low-dimensional (e.g. vectors with length <=1000), dense (so not a lot of zeroes) representation of our texts that we want to work on. For example, the dimensionality reduction functions `pca, nmf, tsne` all take a high-dimensional representation of our text (in the form of a _DocumentTermDF_ (see below) or _VectorSeries_, and return a low-dimensional representation of our text in the form of a _VectorSeries_. +In a _VectorSeries_, every cell is a vector representing text. We use this when we have a low-dimensional (e.g. vectors with length <=1000), dense (so not a lot of zeroes) representation of our texts that we want to work on. For example, the dimensionality reduction functions `pca, nmf, tsne` all take a high-dimensional representation of our text (in the form of a _DataFrame_ (see below) or _VectorSeries_, and return a low-dimensional representation of our text in the form of a _VectorSeries_. -Example of a function that takes as input a _DocumentTermDF_ or _VectorSeries_ and returns a _VectorSeries_: +Example of a function that takes as input a _DataFrame_ or _VectorSeries_ and returns a _VectorSeries_: ```python >>> s = pd.Series(["text first document", "text second document"]).pipe(hero.tokenize).pipe(hero.term_frequency) >>> hero.pca(s) @@ -107,20 +106,18 @@ Example of a function that takes as input a _DocumentTermDF_ or _VectorSeries_ a dtype: object ``` -

DocumentTermDF

+

DataFrame

-As you can see, the `DocumentTermDF` type is a little more complex than the others. Let's have a closer look to see what it is and why, where and how it is used! +As you can see, the `DataFrame` type is a little more complex than the others. Let's have a closer look to see what it is and why, where and how it is used!

What is it?

-A _DocumentTermDF_ is a Pandas implementation of a [Document Term Matrix](https://en.wikipedia.org/wiki/Document-term_matrix), so the rows are the documents and the columns are the words/terms in all the documents. A _DocumentTermDF_ has multiindexed columns with level one -being the content name (e.g. "tfidf"), and level two being the terms. It could look like this: +A _DataFrame_ can be a Pandas implementation of a [Document Term Matrix](https://en.wikipedia.org/wiki/Document-term_matrix), so the rows are the documents and the columns are the words/terms in all the documents. Other representations are Topic-Term matrices or Document-Topic matrices. In the Topic modeling tutorial you can find a more detailed introcution into those. ```python >>> s = pd.Series(["Sentence one one", "Sentence two"]) >>> t = s.pipe(hero.tokenize).pipe(hero.count) # first tokenize Series, then calculate word count ->>> t - count +>>> t Sentence one two 0 1 2 0 1 1 0 1 @@ -136,7 +133,7 @@ count Sentence Sparse[int64, 0] one Sparse[int64, 0] two Sparse[int64, 0] ``` -That's precisely the reason we use this: Pandas internally does not store the zeros we get when e.g. calculating `hero.count`; so we don't see that the word "two" has zero occurrences in the first sentence, it's not stored. This is a massive advantage when dealing with *big data*: In a _DocumentTermDF_, we only store the data that's relevant for each document to save lots and lots of time and space! +That's precisely the reason we use this: Pandas internally does not store the zeros we get when e.g. calculating `hero.count`; so we don't see that the word "two" has zero occurrences in the first sentence, it's not stored. This is a massive advantage when dealing with *big data*: In a _DataFrame_, we only store the data that's relevant for each document to save lots and lots of time and space! Let's look at an example with some more data. ```python @@ -145,13 +142,13 @@ Let's look at an example with some more data. >>> data_count.sparse.density 0.012... ``` -We can see that only around 1.2% of our _DocumentTermDF_ `data_count` is filled, so using the sparse DataFrame is saving us a lot of space. +We can see that only around 1.2% of our _DataFrame_ `data_count` is filled, so using the sparse DataFrame is saving us a lot of space.

When and how is it used? Do I have to work with multiindexes?!

-The _DocumentTermDF_ is mostly used internally for performance reasons. For example, as you can see above, the default output from `hero.count` is such a DataFrame, but if you apply e.g. `hero.pca` afterwards, you don't even notice the _DocumentTermDF_: `s.pipe(hero.count).pipe(hero.normalize).pipe(hero.pca)` works just fine; everything is seamlessly integrated in the library. +The _DataFrame_ is mostly used internally for performance reasons. For example, as you can see above, the default output from `hero.count` is such a Pandas DataFrame, but if you apply e.g. `hero.pca` afterwards, you don't even notice the _DataFrame_: `s.pipe(hero.count).pipe(hero.normalize).pipe(hero.pca)` works just fine; everything is seamlessly integrated in the library. -The only thing you _can_ but _should not_ do is store a _DocumentTermDF_ in your dataframe, as the performance is really bad. If you really want to, you can do it like this like this: +The only thing you _can_ but _should not_ do is store a _DataFrame_ in your dataframe, as the performance is really bad. If you really want to, you can do it like this like this: ```python >>> data = pd.read_csv("https://github.com/jbesomi/texthero/raw/master/dataset/bbcsport.csv") >>> data_count = data["text"].pipe(count) From 49224af89ee824a0e080360d1fc8b05f28a5a73d Mon Sep 17 00:00:00 2001 From: Maximilian Krahn Date: Sat, 5 Sep 2020 18:57:20 +0200 Subject: [PATCH 07/13] fix black issues --- .travis.yml | 2 +- setup.cfg | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index f913f183..c76284b3 100644 --- a/.travis.yml +++ b/.travis.yml @@ -20,7 +20,7 @@ jobs: env: PATH=/c/Python38:/c/Python38/Scripts:$PATH install: - pip3 install --upgrade pip # all three OSes agree about 'pip3' - - pip3 install black + - pip3 install black==19.10b0 - pip3 install ".[dev]" . # 'python' points to Python 2.7 on macOS but points to Python 3.8 on Linux and Windows # 'python3' is a 'command not found' error on Windows but 'py' works on Windows only diff --git a/setup.cfg b/setup.cfg index d6103b02..3f86e7f3 100644 --- a/setup.cfg +++ b/setup.cfg @@ -41,7 +41,7 @@ install_requires = # TODO pick the correct version. [options.extras_require] dev = - black>=19.10b0 + black==19.10b0 pytest>=4.0.0 Sphinx>=3.0.3 sphinx-markdown-builder>=0.5.4 From 52cdc2bbe03423ec9507124226ea1aeeb5a54f1b Mon Sep 17 00:00:00 2001 From: Henri Froese Date: Mon, 14 Sep 2020 18:20:55 +0200 Subject: [PATCH 08/13] Update tutorial with new DataFrame instead of DocumentTermDF Co-authored-by: Maximilian Krahn --- .travis.yml | 2 +- setup.cfg | 2 +- website/docs/getting-started-herotypes.md | 97 ++++++++++------------- 3 files changed, 44 insertions(+), 57 deletions(-) diff --git a/.travis.yml b/.travis.yml index f913f183..c76284b3 100644 --- a/.travis.yml +++ b/.travis.yml @@ -20,7 +20,7 @@ jobs: env: PATH=/c/Python38:/c/Python38/Scripts:$PATH install: - pip3 install --upgrade pip # all three OSes agree about 'pip3' - - pip3 install black + - pip3 install black==19.10b0 - pip3 install ".[dev]" . # 'python' points to Python 2.7 on macOS but points to Python 3.8 on Linux and Windows # 'python3' is a 'command not found' error on Windows but 'py' works on Windows only diff --git a/setup.cfg b/setup.cfg index d6103b02..3f86e7f3 100644 --- a/setup.cfg +++ b/setup.cfg @@ -41,7 +41,7 @@ install_requires = # TODO pick the correct version. [options.extras_require] dev = - black>=19.10b0 + black==19.10b0 pytest>=4.0.0 Sphinx>=3.0.3 sphinx-markdown-builder>=0.5.4 diff --git a/website/docs/getting-started-herotypes.md b/website/docs/getting-started-herotypes.md index c620ce2e..01d2db3d 100644 --- a/website/docs/getting-started-herotypes.md +++ b/website/docs/getting-started-herotypes.md @@ -16,15 +16,15 @@ document_id ``` - Consequently, in the Texthero's _preprocessing_ module, the functions usually take as input a Series where every cell is a string, and return as output a Series where every cell is a string. We will call this kind of Series _TextSeries_, so users know immediately what kind of Series the functions can work on. For example, you might see a function + Consequently, in Texthero's _preprocessing_ module, the functions usually take as input a Series where every cell is a string, and return as output a Series where every cell is a string. We will call this kind of Series _TextSeries_, so users know immediately what kind of Series the functions can work on. For example, you might see a function ```python remove_punctuation(s: TextSeries) -> TextSeries ``` -in the documentation. You then know that this can be used on a DataFrame or Series in the preprocessing phase of your work, where each document is one string. +in the documentation. You then know that this function takes as input a _TextSeries_ and returns as output a _TextSeries_, so it can be used on a DataFrame or Series in the preprocessing phase of your work, where each document is one string.

The four HeroSeries Types

-These are the four types currently supported by the library; almost all of the libraries functions takes as input and return as output one of these types: +These are the three types currently supported by the library; almost all of the libraries functions takes as input and return as output one of these types: 1. **TextSeries**: Every cell is a text, i.e. a string. For example, `pd.Series(["test", "test"])` is a valid TextSeries. @@ -33,36 +33,41 @@ These are the four types currently supported by the library; almost all of the l of strings. For example, `pd.Series([["test"], ["token2", "token3"]])` is a valid TokenSeries. 3. **VectorSeries**: Every cell is a vector representing text, i.e. -a list of floats. For example, `pd.Series([[1.0, 2.0], [3.0]])` is a valid VectorSeries. +a list of floats. For example, `pd.Series([[1.0, 2.0], [3.0, 4.0]])` is a valid VectorSeries. -4. **DocumentTermDF**: A DataFrame where the rows are the documents and the columns are the words/terms in all the documents. The columns are multiindexed with level one -being the content name (e.g. "tfidf"), level two being the individual features and their values. -For example, -`pd.DataFrame([[1, 2, 3], [4,5,6]], columns=pd.MultiIndex.from_tuples([("count", "hi"), ("count", "servus"), ("count", "hola")]))` -is a valid RepresentationSeries. +Additionally, sometimes Texthero functions (most that accept a +VectorSeries as input) also accept a Pandas _DataFrame_ +as input that is representing a matrix. Every cell value +is then one entry in the matrix. An example is +`pd.DataFrame([[1, 2, 3], [4, 5, 6]], columns=["word1", "word2", "word3"])`. Now, if you see a function in the documentation that looks like this: ```python -tfidf(s: TokenSeries) -> DocumentTermDF +tfidf(s: TokenSeries) -> DataFrame ``` then you know that the function takes a Pandas Series whose cells are lists of strings (tokens) and will -return a Pandas DataFrame with the individual features. +return a Pandas DataFrame representing a matrix (in this case a [_Document-Term-Matrix_](https://en.wikipedia.org/wiki/Document-term_matrix) ). You might call it like this: ```python >>> import texthero as hero >>> import pandas as pd >>> s = pd.Series(["Text of first document", "Text of second document"]) ->>> s_tfidf = s.pipe(hero.tokenize).pipe(hero.tfidf) +>>> df_tfidf = s.pipe(hero.tokenize).pipe(hero.tfidf) +>>> df_tfidf + + Text document first of second +0 1.0 1.0 1.405465 1.0 0.000000 +1 1.0 1.0 0.000000 1.0 1.405465 ``` And this function: ```python -pca(s: Union[VectorSeries, DocumentTermDF]) -> VectorSeries +pca(s: Union[VectorSeries, DataFrame]) -> VectorSeries ``` -needs a _DocumentTermDF_ or _VectorSeries_ as input and always returns a _VectorSeries_. +needs a _DataFrame_ or _VectorSeries_ as input and always returns a _VectorSeries_.

The Types in Detail

@@ -107,55 +112,37 @@ Example of a function that takes as input a _DocumentTermDF_ or _VectorSeries_ a dtype: object ``` -

DocumentTermDF

- -As you can see, the `DocumentTermDF` type is a little more complex than the others. Let's have a closer look to see what it is and why, where and how it is used! - -

What is it?

- -A _DocumentTermDF_ is a Pandas implementation of a [Document Term Matrix](https://en.wikipedia.org/wiki/Document-term_matrix), so the rows are the documents and the columns are the words/terms in all the documents. A _DocumentTermDF_ has multiindexed columns with level one -being the content name (e.g. "tfidf"), and level two being the terms. It could look like this: +

DataFrame

-```python ->>> s = pd.Series(["Sentence one one", "Sentence two"]) ->>> t = s.pipe(hero.tokenize).pipe(hero.count) # first tokenize Series, then calculate word count ->>> t - count - Sentence one two -0 1 2 0 -1 1 0 1 -``` +In Natural Language Processing, we are often working with matrices that contain information about our dataset. For example, the output of the functions `tfidf`, `count`, and `term_frequency` is a [Document Term Matrix](https://en.wikipedia.org/wiki/Document-term_matrix), i.e. a matrix where each row is one document and each column is one term / word. +We use a Pandas DataFrame for this for two reasons: +1. It looks nice. +2. It can be sparse. -

Why is it used?

+The second reason is worth explaining in more detail: In e.g. a big Document Term Matrix, we might have 10,000 different terms, so 10,000 columns in our DataFrame. Additionally, most documents will only contain a small subset of all the terms. Thus, in each row, there will be lots of zeros in our matrix. This is why we use a [sparse matrix](https://en.wikipedia.org/wiki/Sparse_matrix): A sparse matrix only stores the non-zero fields. And Pandas DataFrames support sparse data, so Texthero users fully profit from the sparseness! -Have a look at the datatypes of `t` from the last code bit: -```python ->>> t.dtypes -count Sentence Sparse[int64, 0] - one Sparse[int64, 0] - two Sparse[int64, 0] -``` -That's precisely the reason we use this: Pandas internally does not store the zeros we get when e.g. calculating `hero.count`; so we don't see that the word "two" has zero occurrences in the first sentence, it's not stored. This is a massive advantage when dealing with *big data*: In a _DocumentTermDF_, we only store the data that's relevant for each document to save lots and lots of time and space! +This is a massive advantage when dealing with *big data*: In a _sparse DataFrame_, we only store the data that's relevant to save lots and lots of time and space! Let's look at an example with some more data. ```python >>> data = pd.read_csv("https://github.com/jbesomi/texthero/raw/master/dataset/bbcsport.csv") >>> data_count = data["text"].pipe(count) +>>> data_count + 000m 00pm 04secs 05m 09secs ... zornotza ztl zuluaga zurich zvonareva +0 0 0 0 0 0 ... 0 0 0 0 0 +1 0 0 0 0 0 ... 0 0 0 0 0 +2 0 0 0 0 0 ... 0 0 0 0 0 +3 0 0 0 0 0 ... 0 0 0 0 0 +4 3 0 0 0 0 ... 0 0 0 0 0 +.. ... ... ... ... ... ... ... ... ... ... ... +732 0 0 0 0 0 ... 0 0 0 0 0 +733 0 0 0 0 0 ... 0 0 0 0 0 +734 0 0 0 0 0 ... 0 0 0 0 0 +735 0 0 0 0 0 ... 0 0 0 0 0 +736 0 0 0 0 0 ... 0 0 0 0 0 + >>> data_count.sparse.density -0.012... +0.010792808715706939 ``` -We can see that only around 1.2% of our _DocumentTermDF_ `data_count` is filled, so using the sparse DataFrame is saving us a lot of space. - -

When and how is it used? Do I have to work with multiindexes?!

- -The _DocumentTermDF_ is mostly used internally for performance reasons. For example, as you can see above, the default output from `hero.count` is such a DataFrame, but if you apply e.g. `hero.pca` afterwards, you don't even notice the _DocumentTermDF_: `s.pipe(hero.count).pipe(hero.normalize).pipe(hero.pca)` works just fine; everything is seamlessly integrated in the library. - -The only thing you _can_ but _should not_ do is store a _DocumentTermDF_ in your dataframe, as the performance is really bad. If you really want to, you can do it like this like this: -```python ->>> data = pd.read_csv("https://github.com/jbesomi/texthero/raw/master/dataset/bbcsport.csv") ->>> data_count = data["text"].pipe(count) - ->>> # NOT recommended as performance is not optimal ->>> data["count"] = data_count -``` \ No newline at end of file +We can see that only around 1% of our DataFrame `data_count` is filled with non-zero values, so using the sparse DataFrame is saving us a lot of space. From 300b11dd7f68f09f0a70865e0c2faaf958b01918 Mon Sep 17 00:00:00 2001 From: Henri Froese Date: Mon, 14 Sep 2020 18:50:02 +0200 Subject: [PATCH 09/13] incorporate suggested changes II --- website/docs/getting-started-herotypes.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/docs/getting-started-herotypes.md b/website/docs/getting-started-herotypes.md index 88b96f36..1429a12e 100644 --- a/website/docs/getting-started-herotypes.md +++ b/website/docs/getting-started-herotypes.md @@ -20,7 +20,7 @@ document_id ```python remove_punctuation(s: TextSeries) -> TextSeries ``` -in the documentation. You then know that this function takes as input a _TextSeries_ and returns as output a _TextSeries_, so it can be used on a DataFrame or Series in the preprocessing phase of your work, where each document is one string. +in the documentation. You then know that this function takes as input a _TextSeries_ and returns as output a _TextSeries_, so it can be used in the preprocessing phase of your work, where each document is one string.

The four HeroSeries Types

@@ -114,15 +114,15 @@ dtype: object

DataFrame

-In Natural Language Processing, we are often working with matrices that contain information about our dataset. For example, the output of the functions `tfidf`, `count`, and `term_frequency` is a [Document Term Matrix](https://en.wikipedia.org/wiki/Document-term_matrix), i.e. a matrix where each row is one document and each column is one term / word. +In Natural Language Processing, we are often working with matrices that contain information about our dataset. For example, the output of the functions `tfidf`, `count`, and `term_frequency` is a [Document-Term Matrix](https://en.wikipedia.org/wiki/Document-term_matrix), i.e. a matrix where each row is one document and each column is one term / word. We use a Pandas DataFrame for this for two reasons: 1. It looks nice. 2. It can be sparse. -The second reason is worth explaining in more detail: In e.g. a big Document Term Matrix, we might have 10,000 different terms, so 10,000 columns in our DataFrame. Additionally, most documents will only contain a small subset of all the terms. Thus, in each row, there will be lots of zeros in our matrix. This is why we use a [sparse matrix](https://en.wikipedia.org/wiki/Sparse_matrix): A sparse matrix only stores the non-zero fields. And Pandas DataFrames support sparse data, so Texthero users fully profit from the sparseness! +The second reason is worth explaining in more detail: In e.g. a big Document-Term Matrix, we might have 10,000 different terms, so 10,000 columns in our DataFrame. Additionally, most documents will only contain a small subset of all the terms. Thus, in each row, there will be lots of zeros in our matrix. This is why we use a [sparse matrix](https://en.wikipedia.org/wiki/Sparse_matrix): A sparse matrix only stores the non-zero fields. And Pandas DataFrames support sparse data, so Texthero users fully profit from the sparseness! -This is a massive advantage when dealing with *big data*: In a _sparse DataFrame_, we only store the data that's relevant to save lots and lots of time and space! +This is a massive advantage when dealing with *big datasets*: In a _sparse DataFrame_, we only store the data that's relevant to save lots and lots of time and space! Let's look at an example with some more data. ```python From 355fa518aa6422fd804e11709094535630ffd24a Mon Sep 17 00:00:00 2001 From: Henri Froese Date: Tue, 22 Sep 2020 19:50:17 +0200 Subject: [PATCH 10/13] - --- website/docs/getting-started-herotypes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/getting-started-herotypes.md b/website/docs/getting-started-herotypes.md index 1429a12e..c2317a98 100644 --- a/website/docs/getting-started-herotypes.md +++ b/website/docs/getting-started-herotypes.md @@ -22,7 +22,7 @@ remove_punctuation(s: TextSeries) -> TextSeries ``` in the documentation. You then know that this function takes as input a _TextSeries_ and returns as output a _TextSeries_, so it can be used in the preprocessing phase of your work, where each document is one string. -

The four HeroSeries Types

+

The HeroSeries Types

These are the three types currently supported by the library; almost all of the libraries functions takes as input and return as output one of these types: From 1a8715980e6b54b17b4a72caa8eed860be5160b5 Mon Sep 17 00:00:00 2001 From: Henri Froese Date: Tue, 22 Sep 2020 19:57:48 +0200 Subject: [PATCH 11/13] final small tweaks --- website/docs/getting-started-herotypes.md | 26 +++++++++++------------ 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/website/docs/getting-started-herotypes.md b/website/docs/getting-started-herotypes.md index c2317a98..4f500a67 100644 --- a/website/docs/getting-started-herotypes.md +++ b/website/docs/getting-started-herotypes.md @@ -127,20 +127,20 @@ This is a massive advantage when dealing with *big datasets*: In a _sparse DataF Let's look at an example with some more data. ```python >>> data = pd.read_csv("https://github.com/jbesomi/texthero/raw/master/dataset/bbcsport.csv") ->>> data_count = data["text"].pipe(count) +>>> data_count = data["text"].pipe(hero.count) >>> data_count - 000m 00pm 04secs 05m 09secs ... zornotza ztl zuluaga zurich zvonareva -0 0 0 0 0 0 ... 0 0 0 0 0 -1 0 0 0 0 0 ... 0 0 0 0 0 -2 0 0 0 0 0 ... 0 0 0 0 0 -3 0 0 0 0 0 ... 0 0 0 0 0 -4 3 0 0 0 0 ... 0 0 0 0 0 -.. ... ... ... ... ... ... ... ... ... ... ... -732 0 0 0 0 0 ... 0 0 0 0 0 -733 0 0 0 0 0 ... 0 0 0 0 0 -734 0 0 0 0 0 ... 0 0 0 0 0 -735 0 0 0 0 0 ... 0 0 0 0 0 -736 0 0 0 0 0 ... 0 0 0 0 0 + ! " "' ", # $ % ... £62m £6m £70m £7m £7million £80,000 £8m +0 0 5 0 0 0 0 0 ... 0 0 0 0 0 0 0 +1 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 +2 0 14 0 0 0 0 0 ... 0 0 0 0 0 0 0 +3 0 10 0 0 0 0 0 ... 0 0 0 0 0 0 0 +4 0 4 0 0 0 0 0 ... 0 0 0 0 0 0 0 +.. .. .. .. .. .. .. .. ... ... ... ... ... ... ... ... +732 0 2 0 0 0 0 2 ... 0 0 0 0 0 0 0 +733 0 6 0 0 0 0 0 ... 0 0 0 0 0 0 0 +734 0 5 0 0 0 0 0 ... 0 0 0 0 0 0 0 +735 0 14 0 0 0 0 0 ... 0 0 0 0 0 0 0 +736 0 6 0 0 0 0 0 ... 0 0 0 0 0 0 0 >>> data_count.sparse.density 0.010792808715706939 From 98f955e6c8a6422331be1bc7e333214811ed87fc Mon Sep 17 00:00:00 2001 From: Maximilian Krahn Date: Tue, 22 Sep 2020 22:09:20 +0200 Subject: [PATCH 12/13] added tutorial to sidebar --- website/docs/getting-started-herotypes.md | 5 +++++ website/sidebars.json | 3 ++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/website/docs/getting-started-herotypes.md b/website/docs/getting-started-herotypes.md index 4f500a67..f3d98d96 100644 --- a/website/docs/getting-started-herotypes.md +++ b/website/docs/getting-started-herotypes.md @@ -1,3 +1,8 @@ +--- +id: getting-started-herotypes +title: Getting started HeroTypes +--- +

HeroTypes

In Texthero, we're always working with Pandas Series and Pandas Dataframes to gain insights from text data! To make things easier and more intuitive, we differentiate between different types of Series/DataFrames, depending on where we are on the road to understanding our dataset. diff --git a/website/sidebars.json b/website/sidebars.json index bca3356b..6e006c10 100644 --- a/website/sidebars.json +++ b/website/sidebars.json @@ -1,7 +1,8 @@ { "docs": { "Getting Started": [ - "getting-started" + "getting-started", + "getting-started-herotypes" ] }, "api": { From 910d35c486d144dbfeb77a81a77a49877a185ea7 Mon Sep 17 00:00:00 2001 From: Maximilian Krahn Date: Tue, 22 Sep 2020 22:12:49 +0200 Subject: [PATCH 13/13] fixed title --- website/docs/getting-started-herotypes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/getting-started-herotypes.md b/website/docs/getting-started-herotypes.md index f3d98d96..b9882854 100644 --- a/website/docs/getting-started-herotypes.md +++ b/website/docs/getting-started-herotypes.md @@ -1,6 +1,6 @@ --- id: getting-started-herotypes -title: Getting started HeroTypes +title: Getting started - HeroTypes ---

HeroTypes