efort
/
glm4
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix openai_api_server request_id issue 

add request_id=f"{time.time()}" to Fix API concurrency request issue
This commit is contained in:
Lian Junhong 2024-06-06 12:37:26 +08:00 committed by GitHub
parent 1683d673d2
commit 0b979f8bdb
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
basic_demo/openai_api_server.py
View File

@ -195,7 +195,7 @@ async def generate_stream_glm4(params):
"skip_special_tokens": True,
}
sampling_params = SamplingParams(**params_dict)
async for output in engine.generate(inputs=inputs, sampling_params=sampling_params, request_id="glm-4-9b"):
async for output in engine.generate(inputs=inputs, sampling_params=sampling_params, request_id=f"{time.time()}"):
output_len = len(output.outputs[0].token_ids)
input_len = len(output.prompt_token_ids)
ret = {