Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update for Prism V3 #50

Merged
merged 109 commits into from
Nov 6, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
109 commits
Select commit Hold shift + click to select a range
557efe9
Initial commit of new directory.
wd-mgreynolds Aug 26, 2023
45be542
Prism V3 first commit
wd-mgreynolds Aug 26, 2023
fc8f6f0
Prism V3 update
wd-mgreynolds Aug 26, 2023
28c18bc
Prism V3 additional requirements.
wd-mgreynolds Aug 26, 2023
e31a4cb
Update .gitignore
wd-mgreynolds Aug 26, 2023
ad955a3
Update for QA testing
wd-mgreynolds Sep 15, 2023
8f7cd91
Create tables-help.sh
wd-mgreynolds Sep 15, 2023
2243cc3
Updates to handle .gz files.
wd-mgreynolds Sep 16, 2023
1616182
Fixes to bucket create and bucket upload.
wd-mgreynolds Sep 17, 2023
45692f7
Fixes to bucket upload.
wd-mgreynolds Sep 17, 2023
e158802
Fixes to dataChanges list
wd-mgreynolds Sep 17, 2023
37448ea
Fixes to dataChanges run to return activity_id
wd-mgreynolds Sep 17, 2023
635a866
Added dataChanges activities command.
wd-mgreynolds Sep 17, 2023
2e5210b
Added status flag to dataChanges activities
wd-mgreynolds Sep 17, 2023
5e7cef0
Bug fix status flag in dataChanges activities
wd-mgreynolds Sep 17, 2023
89709a6
General bug fixes - remove gzip to disk
wd-mgreynolds Oct 10, 2023
d834993
General bug fixes - remove gzip to disk
wd-mgreynolds Oct 18, 2023
74dc4b3
First release of Prism V3
wd-mgreynolds Oct 18, 2023
cd21a03
Clean-up of DataChanges commands.
wd-mgreynolds Oct 19, 2023
3e71d7a
Merge remote-tracking branch 'origin/master'
wd-mgreynolds Oct 19, 2023
f3c5401
Add/enhance WQL - select * expansion
wd-mgreynolds Oct 20, 2023
a8462be
Docstring updates
wd-mgreynolds Oct 20, 2023
deca3a6
Updates for logging and install.
wd-mgreynolds Oct 21, 2023
7c7d464
Updates for install.
wd-mgreynolds Oct 21, 2023
43247b1
Updates to command imports.
wd-mgreynolds Oct 21, 2023
4f9ce60
Updates to command imports.
wd-mgreynolds Oct 21, 2023
196776e
Downgrade to match 3.9.
wd-mgreynolds Oct 21, 2023
c7c36d5
Change entry point to cli.
wd-mgreynolds Oct 21, 2023
2d1d45b
Tables get - fixed case-insensitive search.
wd-mgreynolds Oct 21, 2023
d5f0b36
Update table commands to use isName nomenclature.
wd-mgreynolds Oct 22, 2023
1dc0336
Update table commands to use tabular nomenclature.
wd-mgreynolds Oct 23, 2023
a220df8
Set output to stdout for INFO logging.
wd-mgreynolds Oct 23, 2023
be29d71
Update patch docstring.
wd-mgreynolds Oct 23, 2023
f085f15
Bug fixes to put:/tables update.
wd-mgreynolds Oct 23, 2023
d1a75c3
Remove ID test from schema_fixup.
wd-mgreynolds Oct 23, 2023
4ba7f8b
Bug fixes to tables_upload.
wd-mgreynolds Oct 23, 2023
66820bd
Bug fixes to buckets_complete to handle http 400 errors.
wd-mgreynolds Oct 23, 2023
8065034
Update to tables upload to include bucket info in result.
wd-mgreynolds Oct 23, 2023
f53a317
Updated schema_fixup to remove tenant specific attributes.
wd-mgreynolds Oct 23, 2023
edccda2
Bug fix tables_get.
wd-mgreynolds Oct 23, 2023
6b6a9f8
Remove get_fields, migrated to prism.schema_fixup.
wd-mgreynolds Oct 23, 2023
6c34927
Bug fix truncate_tables.
wd-mgreynolds Oct 23, 2023
b6e9ebb
Bug fix bucket_files upload of empty file.
wd-mgreynolds Oct 23, 2023
f508910
Bug fix file_containers_load upload of empty file.
wd-mgreynolds Oct 23, 2023
150b6ec
Update for Python 3.9+ and bucket get fixes.
wd-mgreynolds Oct 23, 2023
03f39d3
Remove Python 3.7 and 3.8
wd-mgreynolds Oct 23, 2023
8356ceb
Remove WQL and RaaS commands.
wd-mgreynolds Oct 24, 2023
ac883e7
Merge remote-tracking branch 'origin/master'
wd-mgreynolds Oct 24, 2023
3bcf7e9
Remove Pandas dependency.
wd-mgreynolds Oct 24, 2023
89caf74
Restored Python 3.8
wd-mgreynolds Oct 24, 2023
cfc1081
Replace format option with compact.
wd-mgreynolds Oct 24, 2023
c8b9bd3
Replace schema_fixup with schema_compact.
wd-mgreynolds Oct 24, 2023
915c7ed
Remove --format option.
wd-mgreynolds Oct 24, 2023
3a58a7d
Remove --format option.
wd-mgreynolds Oct 24, 2023
59f824d
Remove RaaS and WQL methods.
wd-mgreynolds Oct 24, 2023
33d9996
Add table_upload_files utility function.
wd-mgreynolds Oct 24, 2023
981e9c6
Added resolve_schema utility function.
wd-mgreynolds Oct 24, 2023
98f2e63
Rename tables_put to tables_edit.
wd-mgreynolds Oct 24, 2023
cd2e717
Bug fix compact_schema.
wd-mgreynolds Oct 24, 2023
6b6a287
Clean-up of --isName option.
wd-mgreynolds Oct 24, 2023
4204100
Buckets get use bucket_id and bucket_name.
wd-mgreynolds Oct 24, 2023
8c3127b
Buckets get bug fix.
wd-mgreynolds Oct 24, 2023
a4a566a
Replace id with bucket_id.
wd-mgreynolds Oct 24, 2023
2a3c814
Replace id/name with table_id/table_name.
wd-mgreynolds Oct 24, 2023
b9d870f
Replace id/name with table_id/table_name.
wd-mgreynolds Oct 24, 2023
00428b3
Fixup on table commands and added truncate_table to prism.
wd-mgreynolds Oct 24, 2023
b1976d5
Bug fix table_name in tables_get by name
wd-mgreynolds Oct 24, 2023
b0b67b2
Remove username and password paramenters.
wd-mgreynolds Oct 24, 2023
9d61ee8
Fixed bucket post messages.
wd-mgreynolds Oct 24, 2023
80134a1
Fixed bug in tables_upload to pass file list.
wd-mgreynolds Oct 24, 2023
047cad8
Fixed bug in tables_upload to pass file list.
wd-mgreynolds Oct 24, 2023
70ebd03
Updated datachange and filecontainers methods to use datachange_id an…
wd-mgreynolds Oct 24, 2023
9004bdb
Bug fix dataChanges_get.
wd-mgreynolds Oct 24, 2023
723bf2c
Bug fix dataChanges get.
wd-mgreynolds Oct 24, 2023
737c4d6
Bug fix dataChanges get case-insensitive search.
wd-mgreynolds Oct 24, 2023
86a3bb4
Bug fix dataChanges validate.
wd-mgreynolds Oct 24, 2023
69ceea4
Bug fix dataChanges validate.
wd-mgreynolds Oct 24, 2023
af5c7a2
Reformat to PEP8 using Black.
wd-mgreynolds Oct 25, 2023
43f8d8f
Reformat to PEP8 using Black.
wd-mgreynolds Oct 25, 2023
e6bb737
Set default bucket name prefix to "prism_python_".
wd-mgreynolds Oct 25, 2023
c9bf978
Test setup.py for package_data.
wd-mgreynolds Oct 25, 2023
02ee979
Replace package_data commands/* package.
wd-mgreynolds Oct 25, 2023
51974d9
Revise table searching limits/offset handling
wd-mgreynolds Oct 26, 2023
b94b9d9
Updates for flake8 and pytest workflow steps.
wd-mgreynolds Oct 26, 2023
02a7daa
Black re-format.
wd-mgreynolds Oct 26, 2023
89cc57d
Added prism.commands to packages - removed from package_data.
wd-mgreynolds Oct 27, 2023
a597c9a
Added flake8 ignore rules to align with black formatting.
wd-mgreynolds Oct 27, 2023
1117c05
Removed username/password parameters.
wd-mgreynolds Oct 27, 2023
6655761
Fixup for README content and examples.
wd-mgreynolds Oct 29, 2023
8cf8165
Update --table_name required message.
wd-mgreynolds Oct 29, 2023
721c607
Fix data types for numeric fields.
wd-mgreynolds Oct 29, 2023
1cd582d
Fix default displayName to avoid duplicates.
wd-mgreynolds Oct 29, 2023
d1596e6
Fix tables_patch command options.
wd-mgreynolds Oct 29, 2023
f1d9c88
Improve tables_patch attribute message.
wd-mgreynolds Oct 30, 2023
a79d044
Updated README.
wd-mgreynolds Oct 31, 2023
2fb1688
Updated README.
wd-mgreynolds Nov 1, 2023
e2c8fd7
Updated README.
wd-mgreynolds Nov 2, 2023
e3f2429
Remove docs for now
CurtLH Nov 4, 2023
7b35577
Simplify README
CurtLH Nov 4, 2023
b376285
Add authors to setup.py
CurtLH Nov 4, 2023
f070283
Remove examples
CurtLH Nov 6, 2023
684ac8f
Fix typo for docstring of upload_file()
CurtLH Nov 6, 2023
919492a
Add tables_create()
CurtLH Nov 6, 2023
2b55d34
Update README example to install specific version
CurtLH Nov 6, 2023
709f700
Merge pull request #1 from wd-mgreynolds/curtlh-patch-01
CurtLH Nov 6, 2023
5ad16f3
Format using Black
CurtLH Nov 6, 2023
b1272ec
Merge pull request #2 from wd-mgreynolds/curtlh-patch-02
CurtLH Nov 6, 2023
2c94830
Reformat with latest version of Black
CurtLH Nov 6, 2023
67ac6db
Merge pull request #3 from wd-mgreynolds/curtlh-patch-02
CurtLH Nov 6, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/pythonpackage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.7, 3.8, 3.9]
python-version: [3.8, 3.9]

steps:
- uses: actions/checkout@v2
Expand Down
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
# Pycharm
.idea/

# MacOS - Desktop Services Store file(s)
.DS_Store

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
Expand All @@ -12,6 +15,7 @@ __pycache__/
# Distribution / packaging
.Python
env/
venv/
build/
develop-eggs/
dist/
Expand Down
111 changes: 62 additions & 49 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,49 +3,62 @@
# Prism-Python

Python client library and command line interface (CLI) for interacting with
Workday’s Prism API V2.
Workday’s Prism API V3.

Workday provides the Prism Analytics REST API web service to work with
Workday Prism Analytics tables, data change tasks, and datasets. You can develop
a software program that uses the different REST endpoints to
programmatically create Prism Analytics tables and datasets and modify
data in them.

The Python **client library** is a REST API wrapper managing the HTTP methods,
URL endpoints and the data required by specific Workday Prism Analytics API
REST operations. Using this client library in Python projects simplifies interactions
with the Workday Prism Analytics REST API providing the rich functionality
of Workday Prism Analytics to your Python project.

The **CLI** is a powerful tool for interacting with a Workday Prism Analytics
REST API client library, allowing you to quickly and easily perform Workday
Prism Analytics tasks from any command line.

## Install
You may install the latest version directly from GitHub with:

To automatically retrieve and install the latest version of this
package directly GitHub, use the following command:

```bash
pip install git+https://github.com/Workday/prism-python.git
$ pip install git+https://github.com/Workday/prism-python.git
```

It is also possible to install a specific tagged release with:

```bash
pip install git+https://github.com/Workday/prism-python.git@0.2.0
$ pip install git+https://github.com/Workday/prism-python.git@0.3.0
```

## Requirements

1. [Register a Workday Prism Analytics API Client.](https://doc.workday.com/reader/J1YvI9CYZUWl1U7_PSHyHA/qAugF2pRAGtECVLHKdMO_A)
Workday Prism Analytics REST APIs use OAuth authentication and the Workday
configurable security model to authorize Workday Prism Analytics operations
in end-user applications. The Workday Prism REST APIs act on behalf of
a Workday user using the client. The user's security profile affects the
REST API access to Workday resources.

In Workday, register an integrations API client with Prism Analytics as its
scope. Obtain the Client ID, Client Secret, and Refresh Token values that the
Prism class requires as parameters.
The Prism client library, and by extension the CLI, require API Client
credentials setup in the target Workday tenant. The API Client credentials
authorize programmatic access to the Workday tenant and provides the identity
of the Workday user to enforce security for all operations.

2. [Obtain the Workday REST API Endpoint.](https://doc.workday.com/reader/J1YvI9CYZUWl1U7_PSHyHA/L_RKkfJI6bKu1M2~_mfesQ)
1. [Register a Workday Prism Analytics API Client.](https://doc.workday.com/admin-guide/en-us/workday-studio/integration-design/common-components/the-prismanalytics-subassembly/tzr1533120600898.html)
2. [Create Refresh Token](https://doc.workday.com/reader/J1YvI9CYZUWl1U7_PSHyHA/L_RKkfJI6bKu1M2~_mfesQ)
3. [Obtain the Workday REST API Endpoint.](https://doc.workday.com/reader/J1YvI9CYZUWl1U7_PSHyHA/L_RKkfJI6bKu1M2~_mfesQ)

In Workday, obtain the Workday REST API endpoint that the Prism class requires
as a parameter.

3. For ease of use, set the following environment variables using the values obtained above:

```bash
export workday_base_url=<INSERT WORKDAY BASE URL HERE>
export workday_tenant_name=<INSERT WORKDAY TENANT NAME HERE>
export prism_client_id=<INERT PRISM CLIENT ID HERE>
export prism_client_secret=<INSERT PRISM CLIENT SECRET HERE>
export prism_refresh_token=<INSERT PRISM REFRESH TOKEN HERE>
```

## Python Example

### Create a new table with Prism API Version 2
### Create a new table with Prism API Version 3

```python
```{python}
import os
import prism

Expand All @@ -55,50 +68,50 @@ p = prism.Prism(
os.getenv("workday_tenant_name"),
os.getenv("prism_client_id"),
os.getenv("prism_client_secret"),
os.getenv("prism_refresh_token"),
version="v2"
os.getenv("prism_refresh_token")
)

# read in your table schema
schema = prism.load_schema("/path/to/schema.json")

# create an empty API table with your schema
table = prism.create_table(p, "my_new_table", schema=schema["fields"])
# create a new table based on the schema.json file
table = prism.tables_create(
p,
table_name="my_new_table",
file="/path/to/schema.json"
)

# print details about new table
# print JSON response body describing the new table.
print(table)
```

### Manage data in an existing table with Prism API Version 2
### Manage data in an existing table with Prism API Version 3
Table Operations Available: `TruncateandInsert`, `Insert`, `Update`, `Upsert`,
`Delete`.

To use the `Update`, `Upsert`, or `Delete` operations, you must specify an
external id field within your table schema.

```python
# upload GZIP CSV file to your table
prism.upload_file(p, "/path/to/file.csv.gz", table["id"], operation="TruncateandInsert")
```{python}
prism.upload_file(
p,
file="/path/to/data.csv.gz",
table_id=table["id"],
operation="Insert"
)
```

## CLI Example

The command line interface (CLI) provides another way to interact with the Prism API.
The CLI expects your credentials to be stored as environment variables, but they can
also be passed into the CLI manually through the use of optional arguments.

```bash
# get help with the CLI
prism --help
$ prism --help

# get help for the tables command
$ prism tables --help

# list the Prism API tables that you have access to
prism list
# list Prism tables you have access to.
$ prism tables get

# create a new Prism API table
prism create my_new_table /path/to/schema.json
# create a new Prism table
$ prism tables create my_new_table /path/to/schema.json

# upload data to a Prism API table
prism upload /path/to/file.csv.gz bbab30e3018b01a723524ce18010811b
# upload data to the new table
$ prism tables upload 83dd72bd7b911000ca2d790e719a0000 /path/to/file1.csv.gz
```

## Bugs
Expand Down
8 changes: 4 additions & 4 deletions environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ name: prism-dev
channels:
- defaults
dependencies:
- python=3.7
- click>=6,<8
- requests>=2.20.0,<3
- python=3.8
- click
- requests
- pytest
- black=19.10b0
- black
- flake8
25 changes: 0 additions & 25 deletions examples/1_extract_data.py

This file was deleted.

94 changes: 0 additions & 94 deletions examples/2_predict_topics.py

This file was deleted.

20 changes: 0 additions & 20 deletions examples/3_upload_predictions.py

This file was deleted.

23 changes: 0 additions & 23 deletions examples/README.md

This file was deleted.

13 changes: 0 additions & 13 deletions examples/environment.yml

This file was deleted.

Loading
Loading