working on moving towards md towncrier support

THREEPOEMS · Jul 16, 2020 · c8d2bcc · c8d2bcc
1 parent 43a6dd6
commit c8d2bcc
Show file tree

Hide file tree

Showing 98 changed files with 265 additions and 2,100 deletions.
diff --git a/.github/workflows/continous-integration.yml b/.github/workflows/continous-integration.yml
@@ -257,9 +257,7 @@ jobs:
         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
       run: |
         GITHUB_TAG=${GITHUB_TAG/refs\/tags\//}
-        sudo apt-get update
-        sudo apt-get -y install pandoc
-        pip install -U github3.py pypandoc
+        pip install -U github3.py
         python3 scripts/publish_gh_release_notes.py
         ./scripts/ping_slack_about_package_release.sh
 

diff --git a/.typo-ci.yml b/.typo-ci.yml
@@ -23,7 +23,6 @@ excluded_files:
   - "*.html"
   - "*.json"
   - "*.md"
-  - "CHANGELOG.rst"
   - "CODE_OF_CONDUCT.md"
   - "CONTRIBUTING.md"
   - "Dockerfile"

diff --git a/CHANGELOG.rst b/CHANGELOG.rst
diff --git a/changelog/4441.doc.md b/changelog/4441.doc.md
@@ -0,0 +1 @@
+Added documentation on `ambiguity_threshold` parameter in Fallback Actions page.
diff --git a/changelog/4441.doc.rst b/changelog/4441.doc.rst
diff --git a/changelog/4536.improvement.rst → changelog/4536.improvement.md b/changelog/4536.improvement.rst → changelog/4536.improvement.md
diff --git a/changelog/4605.doc.rst → changelog/4605.doc.md b/changelog/4605.doc.rst → changelog/4605.doc.md
diff --git a/changelog/4745.feature.md b/changelog/4745.feature.md
@@ -0,0 +1,4 @@
+Added template name to the metadata of bot utterance events.
+
+`BotUttered` event contains a `template_name` property in its metadata for any
+new bot message.
diff --git a/changelog/4745.feature.rst b/changelog/4745.feature.rst
diff --git a/changelog/5038.bugfix.md b/changelog/5038.bugfix.md
@@ -0,0 +1,3 @@
+Fixed a bug in the `CountVectorsFeaturizer` which resulted in the very first
+message after loading a model to be processed incorrectly due to the vocabulary
+not being loaded yet.
diff --git a/changelog/5038.bugfix.rst b/changelog/5038.bugfix.rst
diff --git a/changelog/5086.feature.md b/changelog/5086.feature.md
@@ -0,0 +1,2 @@
+Added a `--num-threads` CLI argument that can be passed to `rasa train`
+and will be used to train NLU components.
diff --git a/changelog/5086.feature.rst b/changelog/5086.feature.rst
diff --git a/changelog/5135.bugfix.rst → changelog/5135.bugfix.md b/changelog/5135.bugfix.rst → changelog/5135.bugfix.md
@@ -1 +1,2 @@
-Fixed Rasa shell skipping button messages if buttons are attached to a message previous to the latest.
+Fixed Rasa shell skipping button messages if buttons are attached to
+a message previous to the latest.
diff --git a/changelog/5385.bugfix.md b/changelog/5385.bugfix.md
@@ -0,0 +1 @@
+Stack level for `FutureWarning` updated to level 2.
diff --git a/changelog/5385.bugfix.rst b/changelog/5385.bugfix.rst
diff --git a/changelog/5453.bugfix.md b/changelog/5453.bugfix.md
@@ -0,0 +1,2 @@
+If custom utter message contains no value or integer value, then it fails
+returning custom utter message. Fixed by converting the template to type string.
diff --git a/changelog/5453.bugfix.rst b/changelog/5453.bugfix.rst
diff --git a/changelog/5481.improvement.md b/changelog/5481.improvement.md
@@ -0,0 +1 @@
+`DIETClassifier` now also assigns a confidence value to entity predictions.
diff --git a/changelog/5481.improvement.rst b/changelog/5481.improvement.rst
diff --git a/changelog/5510.feature.md b/changelog/5510.feature.md
@@ -0,0 +1,46 @@
+You can now define what kind of features should be used by what component
+(see [Choosing a Pipeline](/docs/nlu/choosing-a-pipeline)).
+
+You can set an alias via the option `alias` for every featurizer in your pipeline.
+The `alias` can be anything, by default it is set to the full featurizer class name.
+You can then specify, for example, on the
+[DIETClassifier](/docs/nlu/components#diet-classifier) what features from which
+featurizers should be used.
+If you don't set the option `featurizers` all available features will be used.
+This is also the default behavior.
+Check [Components](/docs/nlu/components) to see what components have the option
+`featurizers` available.
+
+Here is an example pipeline that shows the new option.
+We define an alias for all featurizers in the pipeline.
+All features will be used in the `DIETClassifier`.
+However, the `ResponseSelector` only takes the features from the
+`ConveRTFeaturizer` and the `CountVectorsFeaturizer` (word level).
+
+```
+pipeline:
+- name: ConveRTTokenizer
+- name: ConveRTFeaturizer
+  alias: "convert"
+- name: CountVectorsFeaturizer
+  alias: "cvf_word"
+- name: CountVectorsFeaturizer
+  alias: "cvf_char"
+  analyzer: char_wb
+  min_ngram: 1
+  max_ngram: 4
+- name: RegexFeaturizer
+  alias: "regex"
+- name: LexicalSyntacticFeaturizer
+  alias: "lsf"
+- name: DIETClassifier:
+- name: ResponseSelector
+  epochs: 50
+  featurizers: ["convert", "cvf_word"]
+- name: EntitySynonymMapper
+```
+
+:::caution
+This change is model-breaking. Please retrain your models.
+
+:::
diff --git a/changelog/5510.feature.rst b/changelog/5510.feature.rst
diff --git a/changelog/5617.bugfix.rst → changelog/5617.bugfix.md b/changelog/5617.bugfix.rst → changelog/5617.bugfix.md
diff --git a/changelog/5637.improvement.md b/changelog/5637.improvement.md
@@ -0,0 +1,3 @@
+Added behavior to the `rasa --version` command. It will now also list information
+about the operating system, python version and `rasa-sdk`. This will make it easier
+for users to file bug reports.
diff --git a/changelog/5637.improvement.rst b/changelog/5637.improvement.rst
diff --git a/changelog/5638.bugfix.md b/changelog/5638.bugfix.md
@@ -0,0 +1 @@
+Fixed DIET breaking with empty spaCy model.
diff --git a/changelog/5638.bugfix.rst b/changelog/5638.bugfix.rst
diff --git a/changelog/5640.doc.rst → changelog/5640.doc.md b/changelog/5640.doc.rst → changelog/5640.doc.md
diff --git a/changelog/5675.docs.md b/changelog/5675.docs.md
@@ -0,0 +1,2 @@
+Update `rasa shell` documentation to explain how to recreate external
+channel session behavior.
diff --git a/changelog/5675.docs.rst b/changelog/5675.docs.rst
diff --git a/changelog/5737.bugfix.md b/changelog/5737.bugfix.md
@@ -0,0 +1,4 @@
+Pinned the library version for the Azure
+[Cloud Storage](/docs/user-guide/cloud-storage) to 2.1.0 since the
+persistor is currently not compatible with later versions of the azure-storage-blob
+library.
diff --git a/changelog/5737.bugfix.rst b/changelog/5737.bugfix.rst
diff --git a/changelog/5743.improvement.rst → changelog/5743.improvement.md b/changelog/5743.improvement.rst → changelog/5743.improvement.md
diff --git a/changelog/5748.improvement.rst → changelog/5748.improvement.md b/changelog/5748.improvement.rst → changelog/5748.improvement.md
@@ -1,5 +1,7 @@
-Standardize testing output. The following test output can be produced for intents, responses, entities and stories:
-- report: a detailed report with testing metrics per label (e.g. precision, recall, accuracy, etc.)
+Standardize testing output. The following test output can be produced for intents,
+responses, entities and stories:
+- report: a detailed report with testing metrics per label (e.g. precision,
+  recall, accuracy, etc.)
 - errors: a file that contains incorrect predictions
 - successes: a file that contains correct predictions
 - confusion matrix: plot of confusion matrix

diff --git a/changelog/5755.bugfix.rst → changelog/5755.bugfix.md b/changelog/5755.bugfix.rst → changelog/5755.bugfix.md
@@ -1,2 +1,2 @@
-Remove ``clean_up_entities`` from extractors that extract pre-defined entities.
+Remove `clean_up_entities` from extractors that extract pre-defined entities.
 Just keep the clean up method for entity extractors that extract custom entities.
diff --git a/changelog/5756.improvement.md b/changelog/5756.improvement.md
@@ -0,0 +1,10 @@
+To avoid the problem of our entity extractors predicting entity labels for
+just a part of the words, we introduced a cleaning method after the prediction
+was done. We should avoid the incorrect prediction in the first place.
+To achieve this we will not tokenize words into sub-words anymore.
+We take the mean feature vectors of the sub-words as the feature vector of the word.
+
+:::caution
+This change is model breaking. Please, retrain your models.
+
+:::
diff --git a/changelog/5756.improvement.rst b/changelog/5756.improvement.rst
diff --git a/changelog/5757.removal.md b/changelog/5757.removal.md
@@ -0,0 +1,3 @@
+Removed previously deprecated packages `rasa_nlu` and `rasa_core`.
+
+Use imports from `rasa.core` and `rasa.nlu` instead.
diff --git a/changelog/5757.removal.rst b/changelog/5757.removal.rst
diff --git a/changelog/5758.removal.md b/changelog/5758.removal.md
@@ -0,0 +1,14 @@
+Removed previously deprecated classes:
+- event brokers (`EventChannel` and `FileProducer`, `KafkaProducer`,
+  `PikaProducer`, `SQLProducer`)
+- intent classifier `EmbeddingIntentClassifier`
+- policy `KerasPolicy`
+
+Removed previously deprecated methods:
+- `Agent.handle_channels`
+- `TrackerStore.create_tracker_store`
+
+Removed support for pipeline templates in `config.yml`
+
+Removed deprecated training data keys `entity_examples` and `intent_examples` from
+json training data format.
diff --git a/changelog/5758.removal.rst b/changelog/5758.removal.rst
diff --git a/changelog/5759.improvement.md b/changelog/5759.improvement.md
@@ -0,0 +1,3 @@
+Move option `case_sensitive` from the tokenizers to the featurizers.
+- Remove the option from the `WhitespaceTokenizer` and `ConveRTTokenizer`.
+- Add option `case_sensitive` to the `RegexFeaturizer`.
diff --git a/changelog/5759.improvement.rst b/changelog/5759.improvement.rst
diff --git a/changelog/5788.misc.md b/changelog/5788.misc.md
@@ -0,0 +1,2 @@
+Don't set roles and groups to `O` (nothing found) when constructing entities
+from model predictions.
diff --git a/changelog/5788.misc.rst b/changelog/5788.misc.rst
diff --git a/changelog/5792.bugfix.md b/changelog/5792.bugfix.md
@@ -0,0 +1,2 @@
+Fixed issue where the `DucklingHTTPExtractor` component would
+not work if its `url` contained a trailing slash.
diff --git a/changelog/5792.bugfix.rst b/changelog/5792.bugfix.rst
diff --git a/changelog/5794.improvement.md b/changelog/5794.improvement.md
@@ -0,0 +1,3 @@
+Creating a `Domain` using `Domain.fromDict` can no longer alter the input dictionary.
+Previously, there could be problems when the input dictionary was re-used for other
+things after creating the `Domain` from it.
diff --git a/changelog/5794.improvement.rst b/changelog/5794.improvement.rst
diff --git a/changelog/5805.improvement.md b/changelog/5805.improvement.md
@@ -0,0 +1,4 @@
+The debug-level logs when instantiating an
+[SQLTrackerStore](/docs/api/tracker-stores#sql-tracker-store)
+no longer show the password in plain text. Now, the URL is displayed with the password
+hidden, e.g. `postgresql://username:***@localhost:5432`.
diff --git a/changelog/5805.improvement.rst b/changelog/5805.improvement.rst
diff --git a/changelog/5808.bugfix.md b/changelog/5808.bugfix.md
@@ -0,0 +1 @@
+Changed to variable `CERT_URI` in `hangouts.py` to a string type
diff --git a/changelog/5808.bugfix.rst b/changelog/5808.bugfix.rst
diff --git a/changelog/5811.docs.md b/changelog/5811.docs.md
@@ -0,0 +1 @@
+Event brokers documentation should say `url` instead of `host`.
diff --git a/changelog/5811.docs.rst b/changelog/5811.docs.rst
diff --git a/changelog/5834.removal.md b/changelog/5834.removal.md
@@ -0,0 +1 @@
+Removed `restaurantbot` example as it was confusing and not a great way to build a bot.
diff --git a/changelog/5834.removal.rst b/changelog/5834.removal.rst
diff --git a/changelog/5837.feature.md b/changelog/5837.feature.md
@@ -0,0 +1,2 @@
+Added `--port` commandline argument to the interactive learning mode to allow
+changing the port for the Rasa server running in the background.
diff --git a/changelog/5837.feature.rst b/changelog/5837.feature.rst
diff --git a/changelog/5855.improvement.md b/changelog/5855.improvement.md
@@ -0,0 +1,3 @@
+Shorten the information in tqdm during training ML algorithms based on the log
+level. If you train your model in debug mode, all available metrics will be
+shown during training, otherwise, the information is shorten.
diff --git a/changelog/5855.improvement.rst b/changelog/5855.improvement.rst
diff --git a/changelog/5905.bugfix.md b/changelog/5905.bugfix.md
@@ -0,0 +1,3 @@
+Remove option `token_pattern` from `CountVectorsFeaturizer`.
+Instead all tokenizers now have the option `token_pattern`.
+If a regular expression is set, the tokenizer will apply the token pattern.
diff --git a/changelog/5905.bugfix.rst b/changelog/5905.bugfix.rst
diff --git a/changelog/5913.improvement.md b/changelog/5913.improvement.md
@@ -0,0 +1,4 @@
+Ignore conversation test directory `tests/` when importing a project
+using `MultiProjectImporter` and `use_e2e` is `False`.
+Previously, any story data found in a project subdirectory would be imported
+as training data.
diff --git a/changelog/5913.improvement.rst b/changelog/5913.improvement.rst
diff --git a/changelog/5952.doc.md b/changelog/5952.doc.md
@@ -0,0 +1,2 @@
+Update `rasa init` documentation to include `tests/conversation_tests.md`
+in the resulting directory tree.
diff --git a/changelog/5952.doc.rst b/changelog/5952.doc.rst
diff --git a/changelog/5964.bugfix.rst → changelog/5964.bugfix.md b/changelog/5964.bugfix.rst → changelog/5964.bugfix.md
diff --git a/changelog/5996.feature.md b/changelog/5996.feature.md
@@ -0,0 +1,2 @@
+Introduced a new `YAML` format for Core training data and implemented a parser
+for it. Rasa Open Source can now read stories in both `Markdown` and `YAML` format.
diff --git a/changelog/5996.feature.rst b/changelog/5996.feature.rst
diff --git a/changelog/5998.bugfix.md b/changelog/5998.bugfix.md
@@ -0,0 +1 @@
+`WhitespaceTokenizer` does not remove vowel signs in Hindi anymore.
diff --git a/changelog/5998.bugfix.rst b/changelog/5998.bugfix.rst
diff --git a/changelog/6024.improvement.md b/changelog/6024.improvement.md
@@ -0,0 +1,2 @@
+`rasa data split nlu` now makes sure that there is at least one example per
+intent and response in the test data.
diff --git a/changelog/6024.improvement.rst b/changelog/6024.improvement.rst
diff --git a/changelog/6042.bugfix.md b/changelog/6042.bugfix.md
@@ -0,0 +1,2 @@
+Convert entity values coming from `DucklingHTTPExtractor` to string
+during evaluation to avoid mismatches due to different types.
diff --git a/changelog/6042.bugfix.rst b/changelog/6042.bugfix.rst
diff --git a/changelog/6045.improvment.md b/changelog/6045.improvment.md
@@ -0,0 +1,21 @@
+We updated the way how we save and use features in our NLU pipeline.
+
+The message object now has a dedicated field, called `features`, to store the
+features that are generated in the NLU pipeline. We adapted all our featurizers in a
+way that sequence and sentence features are stored independently. This allows us to
+keep different kind of features for the sequence and the sentence. For example, the
+`LexicalSyntacticFeaturizer` does not produce any sentence features anymore as our
+experiments showed that those did not bring any performance gain just quite a lot of
+additional values to store.
+
+We also modified the DIET architecture to process the sequence and sentence
+features independently at first. The features are concatenated just before
+the transformer.
+
+We also removed the `__CLS__` token again. Our Tokenizers will not
+add this token anymore.
+
+:::caution
+This change is model-breaking. Please retrain your models.
+
+:::
diff --git a/changelog/6045.improvment.rst b/changelog/6045.improvment.rst
diff --git a/changelog/6053.bugfix.md b/changelog/6053.bugfix.md
@@ -0,0 +1,3 @@
+Update `FeatureSignature` to store just the feature dimension instead of the
+complete shape. This change fixes the usage of the option `share_hidden_layers`
+in the `DIETClassifier`.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Added documentation on `ambiguity_threshold` parameter in Fallback Actions page.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		Added a `--num-threads` CLI argument that can be passed to `rasa train`
		and will be used to train NLU components.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Stack level for `FutureWarning` updated to level 2.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		If custom utter message contains no value or integer value, then it fails
		returning custom utter message. Fixed by converting the template to type string.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		`DIETClassifier` now also assigns a confidence value to entity predictions.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		Update `rasa shell` documentation to explain how to recreate external
		channel session behavior.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,3 @@
		Removed previously deprecated packages `rasa_nlu` and `rasa_core`.

		Use imports from `rasa.core` and `rasa.nlu` instead.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		Don't set roles and groups to `O` (nothing found) when constructing entities
		from model predictions.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		Fixed issue where the `DucklingHTTPExtractor` component would
		not work if its `url` contained a trailing slash.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Changed to variable `CERT_URI` in `hangouts.py` to a string type