SamuKata
Lyell Hintz
Lyell Hintz

patreon


ControlNet + More Tutorial [ StreamDiffusionTD 0.2.2 ]

Happy to bring an update of StreamDiffusionTD with some fixes + a video covering many things in regards to ControlNet + StreamDiffusion. Hopefully it is both helpful + inspiring. I have a few more tutorials / overviews of new operators in the works that will be posted soon.

Thank you for the support as always !

Get the 0.2.2 NEW TOX here >>> https://www.patreon.com/posts/111459780

0.2.2 Changelog here >>> https://www.patreon.com/posts/111459875

NEW ControlNet Tutorial >>> https://www.youtube.com/watch?v=gS2Am5DFZO4

NEW DotUpdater - See this for updating to new version of my operators in your project [also covered in ControlNet tutorial] >>> https://youtu.be/MpSnq8-a1Go

Like always, please let me know if you run into issues. The troubleshoot forum on the discord and @ me on there is the best / preferred way.

ControlNet + More Tutorial [ StreamDiffusionTD 0.2.2 ]

Comments

Is there any tutorial or post somewhere on how to install other checkpoints or LoRAS to use with Stream diffusion?

Jonatan Ewald

...Loading sd15 ControlNet model: control_v11p_sd21_depth.safetensors Failed to load ControlNet model https://huggingface.co/thibaud/controlnet-sd21/blob/main/control_v11p_sd21_depth.safetensors due to: Trying to set a tensor of shape torch.Size([320, 320]) in "weight" (which has shape torch.Size([320, 320, 1, 1])), this looks incorrect. Traceback (most recent call last): File "D:\AI TOUCH\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 637, in _load_model controlnet = ControlNetModel.from_single_file(controlnet_file, config_file=config_file, use_safetensors=True, local_files_only=clip_exists, torch_dtype=torch.float16).to(device=self.device, dtype=self.dtype) File "D:\AI TOUCH\StreamDiffusion\venv\lib\site-packages\diffusers\loaders\single_file.py", line 617, in from_single_file controlnet = download_controlnet_from_original_ckpt( File "D:\AI TOUCH\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\convert_from_ckpt.py", line 1846, in download_controlnet_from_original_ckpt controlnet = convert_controlnet_checkpoint( File "D:\AI TOUCH\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\convert_from_ckpt.py", line 1125, in convert_controlnet_checkpoint set_module_tensor_to_device(controlnet, param_name, "cpu", value=param) File "D:\AI TOUCH\StreamDiffusion\venv\lib\site-packages\accelerate\utils\modeling.py", line 292, in set_module_tensor_to_device raise ValueError( ValueError: Trying to set a tensor of shape torch.Size([320, 320]) in "weight" (which has shape torch.Size([320, 320, 1, 1])), this looks incorrect.

Shio Visconti

issue, when i start streaming with controlnet ON, show me this error: "ValueError: Trying to set a tensor of shape torch.Size([320, 320]) in "weight" (which has shape torch.Size([320, 320, 1, 1])), this looks incorrect. " help pleaseee

Ignacia .Hurtado

Hello, when start streaming it says in the console : " V2V + ControlNet are not supported and will be disabled. A matching Triton is not available, some optimizations will not be enabled. Error caught was: No module named 'triton'. " Do you have an idea where this issue comes from and how I can resolve it ?

Dji

hey, can you tell me how to do the pose controller? from your video I saw depth and edge but couldn't find any pose related object in TD

Kasım Açıkbaş

I can't do it on streaming. cmd doesn't load data. can you please help me

Raja Singa

yeah Im having problems with this too - earlier versions could

Gus

Did it ever finish loading? maybe restart whole process. Depending on the amount of vram on your gpu, it could be having trouble fitting on it, I would recommend 8gigs at least.

dotsimulate

You can negative prompt certain things with negative weight on LoRA models, which is a bit more limiting but could work for certain things.

dotsimulate

negative prompts don't work with lcm / 1 step processes such as streamdiffusion unfortunately.

dotsimulate

[issue] ...Loading sd-turbo (2.1 distilled) model pipeline... ...Loading sd-turbo (2.1 distilled) model: sd-turbo Loading pipeline components...: 0%| | 0/5 [00:00

Andrea Bucci

how do I enable the additional settings for negative prompts?

Edmond Yang

Thank you so much for the controlnet tutorial. Would you also share how to use openpose in stream diffusion ? How to get the colorful skeleton form video source? Thanks in advance.

Dan Fong

Hi all, I'm enjoying this tool, thanks DotSimulate. But I'd like to know if you have found models better than SD-TURBO; do you have any advice?

Ismael Abrego

What would be the equivalent of img2img in comfy? I get good results when I send a video thru stream diffusion but I would like to increase the quality and fps, and I'm looking for the equivalent with stable diffusion / comfyui. Thanks !

Frédéric Bussière

same issue. fixed with pip install torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cu121 pip install huggingface-hub==0.24.6 numpy==1.26.4 thanks dot!

jeevanpillay

same

TITO FERRARI

I have the same error when I run the stream, Did you manage to solve it?

Diskorsia

lo voy a ver, Gracias Dot!!

Julii Haid

Traceback (most recent call last): File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 683, in _load_model pipe = pipeline_class.from_pretrained( File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1090, in from_pretrained cached_folder = cls.download( File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1856, in download cached_folder = snapshot_download( File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\venv\lib\site-packages\huggingface_hub\utils\_validators.py", line 114, in _inner_fn return fn(*args, **kwargs) File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\venv\lib\site-packages\huggingface_hub\_snapshot_download.py", line 216, in snapshot_download raise LocalEntryNotFoundError( huggingface_hub.errors.LocalEntryNotFoundError: Cannot find an appropriate cached snapshot folder for the specified revision on the local disk and outgoing traffic has been disabled. To enable repo look-ups and downloads online, pass 'local_files_only=False' as input. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\42708\Desktop\StreamDiff\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 687, in _load_model except huggingface_hub.utils._errors.LocalEntryNotFoundError: AttributeError: module 'huggingface_hub.utils' has no attribute '_errors' Failed to load model due to: module 'huggingface_hub.utils' has no attribute '_errors'

Julii Haid

this issue is caused by hugging face having a newer version. Click on .tox file on Manual Activation VENV then type this to downgrade huggingface to the right version : pip install huggingface-hub==0.24.6 Then launch it and it should download dependencies (I had to relaunch twotimes but now its working)

Johao Mikaël Marin Gagné

Why prompt No internet connection detected。

is there only 512x512 ? or is there a way to have a better resolution?

Mariana Kamke

# StreamDiffusionTD huggingface_hub.utils._errors troubleshooting guide 1. Hit "Manual Activate" (install page) pulse to open command window. 2. Check package versions: ``` pip list ``` Look for: - torch: 2.1.0 - huggingface-hub: 0.24.6 - numpy: 1.26.4 3. Check CUDA: ``` python -c "import torch; print(torch.cuda.is_available())" ``` Should print "True". 4. If versions wrong or CUDA unavailable, check CUDA version: ``` nvidia-smi ``` 5. Install correct packages ( AND replace cu121 with cu118 if you have cuda11.8): ``` pip install torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cu121 pip install huggingface-hub==0.24.6 numpy==1.26.4 ``` 6. Recheck CUDA (step 3). 7. If you continue to have an issue, please join the discord as it is way easier to troubleshoot on there. And post in the troubleshooting forum the details from above.

dotsimulate

see above. Also follow up on discord. It is impossible to get notified or keep up with this via patreon comments.

dotsimulate

# StreamDiffusionTD huggingface_hub.utils._errors troubleshooting guide 1. Hit "Manual Activate" (install page) pulse to open command window. 2. Check package versions: ``` pip list ``` Look for: - torch: 2.1.0 - huggingface-hub: 0.24.6 - numpy: 1.26.4 3. Check CUDA: ``` python -c "import torch; print(torch.cuda.is_available())" ``` Should print "True". 4. If versions wrong or CUDA unavailable, check CUDA version: ``` nvidia-smi ``` 5. Install correct packages (replace cu121 if needed): ``` pip install torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cu121 pip install huggingface-hub==0.24.6 numpy==1.26.4 ``` 6. Recheck CUDA (step 3). 7. If you continue to have an issue, please join the discord as it is way easier to troubleshoot on there. And post in the troubleshooting forum the details from above.

dotsimulate

Yes. I am looking at this. It seems to not be installing all of the pip packages it should. The issue started about 5 days ago or so. still tracking down what is causing it but there are a few fixes that have worked for people in the discord. I will summarize below.

dotsimulate

got the same thing trying to use any model other than sdturbo

Sasha Bédard

I'm having the same issue today...

Jack Kalish

hi! i have this issue: ...Loading sd-turbo (2.1 distilled) model pipeline... ...Loading sd-turbo (2.1 distilled) model: sd-turbo Traceback (most recent call last): File "D:\touchAI\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 683, in _load_model pipe = pipeline_class.from_pretrained( File "D:\touchAI\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1090, in from_pretrained cached_folder = cls.download( File "D:\touchAI\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1856, in download cached_folder = snapshot_download( File "D:\touchAI\StreamDiffusion\venv\lib\site-packages\huggingface_hub\utils\_validators.py", line 114, in _inner_fn return fn(*args, **kwargs) File "D:\touchAI\StreamDiffusion\venv\lib\site-packages\huggingface_hub\_snapshot_download.py", line 216, in snapshot_download raise LocalEntryNotFoundError( huggingface_hub.errors.LocalEntryNotFoundError: Cannot find an appropriate cached snapshot folder for the specified revision on the local disk and outgoing traffic has been disabled. To enable repo look-ups and downloads online, pass 'local_files_only=False' as input. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "D:\touchAI\StreamDiffusion\streamdiffusionTD\wrapper_td.py", line 687, in _load_model except huggingface_hub.utils._errors.LocalEntryNotFoundError: AttributeError: module 'huggingface_hub.utils' has no attribute '_errors' Failed to load model due to: module 'huggingface_hub.utils' has no attribute '_errors'

Luca Corsi

Hello. Thank you for the amazing work! does 0.2.2 work on mac? Oh I just saw in changelog that you will update Mac-support. I wanna test that :)

Victor E Morales

Mac install guide?

Stuart Smith

Why can't I download streamDiffusion? The message "no internet connection detected "is always displayed+1

Q 小

Why can't I download streamDiffusion? The message "no internet connection detected "is always displayed

WDH2021

Can you record a tutorial for a plug-in to load the civitai Lora model library? I downloaded several model libraries and loaded them in the models in the plug-in, but the effect I got was similar to the effect of not binding the plug-in. I'm not sure whether my model library is not running now. Is there any way to judge that my model library is being used normally?

fan ze

Amaaazing work man! Very cool stuff. I’m curious, as I have a couple of GPU’s…. would there be a way for 2 frames to be processed at once on say a 4090 and 4080 setup? I’ve seen some chatter on this being the way to leverage them, in lieu of actual multi-gpu crunching on each frame. Appreciate it!

Kevin Friel

hello, im newbie, i already install the streamdiffusion 0.2.2, after install the step 1 and step 2, i try to run start stream, the command prompt stuck in last log here Loading pipeline components...: 0%| | 0/5 [00:00

Yanandra Ardi Putra P

Thanks for the fast fix!! Always appreciated.

Garry

When I click Download StreamDiffusion in Install, the message "No internet connection detected" is displayed.

WDH2021

www.patreon.com/posts/dotupdater-1-0-3-111520883

dotsimulate

ah! wow. I didn't realize that wasn't uploaded.

dotsimulate

Thank you for the great work! Where do you link to the DotUpdater tox?

Al Order

Hey Garry. I have fixed this and reuploaded 0.2.2 Thanks for letting me know.

dotsimulate

[Issues] : I recieve a "Script Error" when I pulse the second Install step (Venv~) By the way, thanks for the great work!! [Edit] : 0.2.0 works well

Garry


More Creators