Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added unittest for structured_compress_prompt and fixed bugs #95

Merged
merged 7 commits into from
Feb 28, 2024

Conversation

SiyunZhao
Copy link
Contributor

No description provided.

@SiyunZhao SiyunZhao added bug Something isn't working feature feature style some problem of code style labels Feb 26, 2024
@SiyunZhao SiyunZhao requested a review from iofu728 February 26, 2024 03:21
@SiyunZhao SiyunZhao self-assigned this Feb 26, 2024
@SiyunZhao SiyunZhao force-pushed the siyun-unittest/LLMLingua branch from ffa8928 to 7f033b9 Compare February 26, 2024 06:21
@@ -953,26 +973,12 @@ def sync_sentence(segments, text):
new_segments_info = []
for s in sentences:
tmp = [jj for ii, jj in enumerate(s) if sentence_flags[idx + ii]]
res.append("\n".join(tmp))
res.append("".join(tmp))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switching to a customizable parameter.

@iofu728
Copy link
Contributor

iofu728 commented Feb 28, 2024

Todo:

  • Switching to a customizable parameter;
  • Add json compress prompt function;
  • Add documents

@iofu728 iofu728 merged commit 9f97ba7 into develop Feb 28, 2024
10 checks passed
@iofu728 iofu728 deleted the siyun-unittest/LLMLingua branch February 28, 2024 04:52
iofu728 added a commit that referenced this pull request Feb 28, 2024
* update the definition of compression ratio, structured prompt compress
* merge part of structured_compress_prompt into compress_prompt
* add no-compression retention to sentence and context level filter
* update global ratio and global target token
* fix segments_info error
* change input paramete from ratio to rate
* change parameter name
* Feature(LLMLingua): add template & test scripts
* Feature(LLMLingua): add pre commit check & unittest
* Feature(LLMLingua): update the release script
* Feature(LLMLingua): add HF_TOKEN
* Added unittest for structured_compress_prompt and fixed bugs (#95)
- fix bugs and add unittests
- make style
- add nltk init
- fix nltk file exist error
- add unittest for different models

---------

Co-authored-by: siyunzhao <siyunzhao@microsoft.com>
Co-authored-by: Qianhui Wu <wuqh_thu@foxmail.com>
Co-authored-by: Xufang Luo <34053802+XufangLuo@users.noreply.github.com>
Co-authored-by: Yuqing Yang <justin.yqyang@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working feature feature style some problem of code style
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants