docs: add dataset schema validation #1304

katacek · 2024-11-27T12:47:48Z

resolves #1295 : adding documentation for dataset validation plus updating related part in monitoring

katacek · 2024-11-27T13:02:59Z

sources/platform/actors/development/actor_definition/dataset_schema/index.md

@@ -118,7 +118,7 @@ The template above defines the configuration for the default dataset output view

 The default behavior of the Output tab UI table is to display all fields from `transformation.fields` in the specified order. You can customize the display properties for specific formats or column labels if needed.

-![Output tab UI](./images/output-schema-example.png)
+![Output tab UI](../images/output-schema-example.png)


the image is used also on some other page so I have left it in the original folder and just change the path, but it can be done either way (change the folder and path for the othe page)

katacek · 2024-11-27T13:03:35Z

sources/platform/monitoring/index.md

@@ -41,12 +41,22 @@ Currently, the monitoring option offers the following features:

 ### Alert configuration

-When you set up an alert, you have two choices for how you want the metrics to be evaluated. And depending on your choices, the alerting system will behave differently:
+When you set up an alert, you have four choices for how you want the metrics to be evaluated. And depending on your choices, the alerting system will behave differently:


as a part of the validation, new monitoring possibilities arised, I have added them here

sources/platform/actors/development/actor_definition/dataset_schema/validation.md

gippy

Small changes needed, but otherwise I think this is hopefully good enough for most users :)

sources/platform/actors/development/actor_definition/dataset_schema/validation.md

…chema/validation.md Co-authored-by: Jaroslav Hejlek <[email protected]>

sources/platform/actors/development/actor_definition/dataset_schema/validation.md

mtrunkat · 2024-12-03T10:14:52Z

sources/platform/actors/development/actor_definition/dataset_schema/validation.md

+
+:::info
+
+The schema defines a single item in the dataset. Be careful not to define the schema as an array, it always needs to be a schema of an object.


@gippy, does user get an error when this happens?

I'm not actually 100% sure. I think you could theoretically set the top level type of the schema to array. Will test it tomorrow. If it's possible then we will try to add some check to build so that the creator cannot do it.

Y, let's throw an error as early as possible so ideally in the build. Later it's too late.

sources/platform/actors/development/actor_definition/dataset_schema/validation.md

mtrunkat · 2024-12-03T10:18:47Z

sources/platform/actors/development/actor_definition/dataset_schema/validation.md

+
+## Dataset validation
+
+When you define a schema of your default dataset, the schema is then always used when you insert data into the dataset to perform validation (we use [AJV](https://ajv.js.org/)).


One thing to consider - do we want to mention we currently use AJV? Perhaps AJV contains some JSON schema extensions, and if we replace them, we could change the expected behavior. Or is this not the thing?

I think it's better to mention it, because the creators can then look up the validation output and what it means. I do not think AJV contains any additions to JSON schema.

mtrunkat · 2024-12-03T10:19:47Z

sources/platform/actors/development/actor_definition/dataset_schema/validation.md

+
+**If the data you attempt to store in the dataset is invalid** (meaning any of the items received by the API fails the validation), **the whole request is discarded** and the API will return a response with status code 400 and the following JSON response:
+
+```json


Please update the API docs with this and link them to this documentation. It's important to have it there.

good catch, thanks, api docs part updated, link added

sources/platform/actors/development/actor_definition/dataset_schema/validation.md

gippy

Approving, but I agree with Mara's comments

…chema/validation.md Co-authored-by: Michał Olender <[email protected]>

…chema/validation.md Co-authored-by: Marek Trunkát <[email protected]>

sources/platform/actors/development/actor_definition/dataset_schema/validation.md

…chema/validation.md Co-authored-by: Michał Olender <[email protected]>

TC-MO

Good to go, spotted two things

Double occurence of separate
Typo

sources/platform/actors/development/actor_definition/dataset_schema/validation.md

…chema/validation.md Co-authored-by: Michał Olender <[email protected]>

docs: add dataset schema validation

9ecd6ef

katacek added the t-console Issues with this label are in the ownership of the console team. label Nov 27, 2024

katacek added this to the 103rd sprint - Console team milestone Nov 27, 2024

katacek self-assigned this Nov 27, 2024

katacek added 4 commits November 27, 2024 13:54

fix lint

b96183f

fix lint

c4a48ec

capitalized Actor

d1b2db6

one two three four

087f164

katacek commented Nov 27, 2024

View reviewed changes

katacek marked this pull request as ready for review November 27, 2024 14:16

katacek requested a review from TC-MO as a code owner November 27, 2024 14:16

gippy reviewed Nov 29, 2024

View reviewed changes

sources/platform/actors/development/actor_definition/dataset_schema/validation.md Outdated Show resolved Hide resolved

remove examples of dataset field statistics

2372e0b

katacek requested a review from gippy November 29, 2024 11:50

gippy requested review from netmilk and mtrunkat December 2, 2024 10:31

gippy requested changes Dec 2, 2024

View reviewed changes

katacek and others added 5 commits December 3, 2024 08:41

Update sources/platform/actors/development/actor_definition/dataset_s…

b5b2032

…chema/validation.md Co-authored-by: Jaroslav Hejlek <[email protected]>

Update sources/platform/actors/development/actor_definition/dataset_s…

f1d989a

…chema/validation.md Co-authored-by: Jaroslav Hejlek <[email protected]>

Update sources/platform/actors/development/actor_definition/dataset_s…

73e0f6c

…chema/validation.md Co-authored-by: Jaroslav Hejlek <[email protected]>

Update sources/platform/actors/development/actor_definition/dataset_s…

4bdf857

…chema/validation.md Co-authored-by: Jaroslav Hejlek <[email protected]>

add link, format fixes

4844192

katacek requested a review from gippy December 3, 2024 08:12

mtrunkat requested changes Dec 3, 2024

View reviewed changes

gippy approved these changes Dec 3, 2024

View reviewed changes

katacek and others added 4 commits December 4, 2024 14:51

Update sources/platform/actors/development/actor_definition/dataset_s…

4aca8a8

…chema/validation.md Co-authored-by: Michał Olender <[email protected]>

Update sources/platform/actors/development/actor_definition/dataset_s…

7356b35

…chema/validation.md Co-authored-by: Marek Trunkát <[email protected]>

Update sources/platform/actors/development/actor_definition/dataset_s…

d484ad0

…chema/validation.md Co-authored-by: Marek Trunkát <[email protected]>

Update sources/platform/actors/development/actor_definition/dataset_s…

1adaf5b

…chema/validation.md Co-authored-by: Marek Trunkát <[email protected]>

katacek added 2 commits December 4, 2024 16:34

Merge branch 'master' into docs/add-dataset-schema-validation

89a4166

merge master plus open api

dc6329e

TC-MO requested changes Dec 4, 2024

View reviewed changes

katacek and others added 12 commits December 4, 2024 17:49

add info to api docs

4fb5eaf

api docs part final

9c6c99c

no subsequent admonitions

0f9872b

Update sources/platform/actors/development/actor_definition/dataset_s…

12142db

…chema/validation.md Co-authored-by: Michał Olender <[email protected]>

Update sources/platform/actors/development/actor_definition/dataset_s…

df06398

…chema/validation.md Co-authored-by: Michał Olender <[email protected]>

Update sources/platform/actors/development/actor_definition/dataset_s…

6d26ea0

…chema/validation.md Co-authored-by: Michał Olender <[email protected]>

Update sources/platform/actors/development/actor_definition/dataset_s…

976dd4f

…chema/validation.md Co-authored-by: Michał Olender <[email protected]>

Update sources/platform/actors/development/actor_definition/dataset_s…

fbb6dcd

…chema/validation.md Co-authored-by: Michał Olender <[email protected]>

Update sources/platform/actors/development/actor_definition/dataset_s…

64b51e0

…chema/validation.md Co-authored-by: Michał Olender <[email protected]>

Update sources/platform/actors/development/actor_definition/dataset_s…

b07d03d

…chema/validation.md Co-authored-by: Michał Olender <[email protected]>

Update sources/platform/actors/development/actor_definition/dataset_s…

310d614

…chema/validation.md Co-authored-by: Michał Olender <[email protected]>

Update sources/platform/actors/development/actor_definition/dataset_s…

c518b6b

…chema/validation.md Co-authored-by: Michał Olender <[email protected]>

katacek requested review from mtrunkat and TC-MO December 5, 2024 11:36

mtrunkat approved these changes Dec 5, 2024

View reviewed changes

katacek mentioned this pull request Dec 5, 2024

Dataset schema validation: add data property to API error object apify/apify-client-python#306

Closed

TC-MO approved these changes Dec 6, 2024

View reviewed changes

sources/platform/actors/development/actor_definition/dataset_schema/validation.md Outdated Show resolved Hide resolved

sources/platform/actors/development/actor_definition/dataset_schema/validation.md Outdated Show resolved Hide resolved

katacek and others added 2 commits December 6, 2024 11:02

Update sources/platform/actors/development/actor_definition/dataset_s…

8e0de86

…chema/validation.md Co-authored-by: Michał Olender <[email protected]>

Update sources/platform/actors/development/actor_definition/dataset_s…

330dfe2

…chema/validation.md Co-authored-by: Michał Olender <[email protected]>

katacek merged commit 6873e01 into master Dec 6, 2024
8 checks passed

katacek deleted the docs/add-dataset-schema-validation branch December 6, 2024 10:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: add dataset schema validation #1304

docs: add dataset schema validation #1304

katacek commented Nov 27, 2024

katacek Nov 27, 2024

katacek Nov 27, 2024

gippy left a comment

mtrunkat Dec 3, 2024

gippy Dec 3, 2024

mtrunkat Dec 4, 2024

mtrunkat Dec 3, 2024

gippy Dec 3, 2024

mtrunkat Dec 3, 2024

katacek Dec 5, 2024

gippy left a comment

TC-MO left a comment


		:::info

		The schema defines a single item in the dataset. Be careful not to define the schema as an array, it always needs to be a schema of an object.


		## Dataset validation

		When you define a schema of your default dataset, the schema is then always used when you insert data into the dataset to perform validation (we use [AJV](https://ajv.js.org/)).


		If the data you attempt to store in the dataset is invalid (meaning any of the items received by the API fails the validation), the whole request is discarded and the API will return a response with status code 400 and the following JSON response:

		```json

docs: add dataset schema validation #1304

docs: add dataset schema validation #1304

Conversation

katacek commented Nov 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gippy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gippy left a comment

Choose a reason for hiding this comment

TC-MO left a comment

Choose a reason for hiding this comment