AI Enhanced Flux
Added 2024-08-07 20:30:02 +0000 UTC
Flux Updates!
Things are moving along nicely in the Flux.1 world and soon we'll have things like controlnets, loras! But, for now at least, we've got image-to-image and LLM assistants. These ComfyUI workflows are similar to the previous ones, but updated to use the standard "KSampler" and "Load Checkpoint" nodes. The checkpoint for this is different - https://huggingface.co/Comfy-Org/flux1-dev/blob/main/flux1-dev-fp8.safetensors - though the reduction in quality isn't too bad.
I should really look into using the image input for LLM party, and it looks like there are so many more things you could do as well!
Links
Examples: https://comfyanonymous.github.io/ComfyUI_examples/flux/
Florence: https://github.com/kijai/ComfyUI-Florence2
LLM Party: https://github.com/heshengtao/comfyui_LLM_party
Ollama: https://github.com/ollama/ollama
Workflows
As always, the workflows used in the video can be found attached to this post!
Comments
The info for ays+ is in the workflow note right next to it. I've pasted it below for reference also: "denoise = 1 for empty latent denoise = <1 (~0.85) for image keep CFG = 1.0 ays+ = https://github.com/pamparamm/ComfyUI-ppm Select empty or img latent" Windows is indeed a bit of a headache when it comes to anything AI! One reason I switched to Linux was for the easy life where everything just works :)
Nerdy Rodent
2024-09-25 18:02:28 +0000 UTC@NerdyRodent I was playing with these and notice that most have a scheduler of ays+. Easy to change away from and run them, but if you think it is a good one I'd like to try it. Where the heck do you get it though? I too have the Florence issues. I tend to use the preview version of VS2022 though and the CUDA stuff seems sensitive to that, so I just pick a different attention model instead.
Trevor Baxter
2024-09-25 08:05:38 +0000 UTCSwitching to sdpa in the florence2 node works fine
blechforelle
2024-09-04 21:31:50 +0000 UTCyes got it thank you
delete_me
2024-08-29 09:40:02 +0000 UTCDo you mean the ComfyUI_yanc Save Image node from https://www.patreon.com/posts/flux-1-lora-110291890 ?
Nerdy Rodent
2024-08-29 07:55:27 +0000 UTChi save file node is undefined. Where do I get that. thanks
delete_me
2024-08-28 22:04:11 +0000 UTCAll 3 options are pretty much the same, so if flash attention isn't installed then you can use either of the other two. Florence doesn't produce any images here - only text captions - so if you're getting black images, check for other errors in your setup. So long as you see the text output from Florence, it's doing it's thing!
Nerdy Rodent
2024-08-27 17:00:43 +0000 UTCa conflicting node is installed and it's using that one instead
Nerdy Rodent
2024-08-27 16:53:25 +0000 UTChow did you fix the black image ? I have same issue as you
ight
2024-08-27 14:32:45 +0000 UTCif i change "attention" to "eager", i get a black image output.. (but no errors)
ight
2024-08-27 13:59:22 +0000 UTCsame here :( im on windows, and i cant get the 2 workflow with "Florence" to work, quite disappointed... I get "FlashAttention2 has been toggled on, but it cannot be used due to the following error: the package flash_attn seems to be not installed. Please refer to the documentation of https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2 to install Flash Attention 2." error , so then I installed Visual Studio C++ and tried to run "pip install flash-attn --no-build-isolation" but i get another error "c:\Users\xxx\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\include\torch/csrc/python_headers.h(12): fatal error C1083: Cannot open include file: 'Python.h': No such file or directory error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.41.34120\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2" and now I am stuck and dont know what to do :(
ight
2024-08-27 13:21:20 +0000 UTCDid you remember to set the switch to empty latent when changing the empty latent batch size? I guess you must have a very high VRAM GPU!
Nerdy Rodent
2024-08-15 17:22:17 +0000 UTCOk and last question for me : Why when i try to put a number greater than 1 in the batch_size, there is only one image that comes out?
ON A LES CROOO
2024-08-15 12:50:13 +0000 UTCThere's barely any difference between the 3 attentions, so just use any :) You're also free to use any Florence model of your choice! You'll need ComfyUI PPM for ays as a non-custom sampler. See the notes in the workflow for more information.
Nerdy Rodent
2024-08-15 10:50:19 +0000 UTCok.. i try to reinstall florence2 and it seems fixed ! Flashattention 2 does not work, even when downloaded : cd hopper python setup.py install I still managed to launch with eager or sdpa . But I can't figure out what changes between the 3 attentions? ( sdpa , eager , flash_attention_2) When I launch, I can get a description of my text, I then try to put the text in cliptextencodeflux to get a similar image: the image I have in output is completely black, and comfyui doesn't seem to indicate an error. Do you know how to solve this problem? Another thing: Why do we use the Florence-2base model and not Florence-2large? Also, how to install ays+ in the scheduler..? THANKS
ON A LES CROOO
2024-08-15 10:18:18 +0000 UTCTry a re-download - https://github.com/kijai/ComfyUI-Florence2/issues/39 - you could also try going for a standard install over the portable download!
Nerdy Rodent
2024-08-15 08:04:48 +0000 UTCHello ! when i start Florence img2img v1 i have this error: Error occurred when executing DownloadAndLoadFlorence2Model: Error no file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory C:\Users\SergioLucas\Desktop\ComfyUI_windows_portable\ComfyUI\models\LLM\Florence-2-large. File "C:\Users\SergioLucas\Desktop\ComfyUI_windows_portable\ComfyUI\execution.py", line 152, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\SergioLucas\Desktop\ComfyUI_windows_portable\ComfyUI\execution.py", line 82, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\SergioLucas\Desktop\ComfyUI_windows_portable\ComfyUI\execution.py", line 75, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\SergioLucas\Desktop\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Florence2\nodes.py", line 90, in loadmodel model = AutoModelForCausalLM.from_pretrained(model_path, attn_implementation=attention, device_map=device, torch_dtype=dtype,trust_remote_code=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\SergioLucas\Desktop\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\auto\auto_factory.py", line 559, in from_pretrained return model_class.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\SergioLucas\Desktop\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\modeling_utils.py", line 3460, in from_pretrained raise EnvironmentError
ON A LES CROOO
2024-08-14 23:22:57 +0000 UTCAha, your ratlike cunning has triumphed! I swapped the YOLO node for one from Derfuu, and that produces the correct(ish) numbers in the maths boxes, and all is right with the world again. Although I still don't understand how the floor function is producing the numbers it does, but that a problem for another day. Many thanks for your patience.
TigerFox57
2024-08-13 20:25:23 +0000 UTCThere you go, that's what it is then. Be sure to use the one specified in the note next to it - GetImageSize = stability-ComfyUI-nodes
Nerdy Rodent
2024-08-13 20:22:11 +0000 UTCYes, it is installed. I went into the manager and disabled it, restarted comfy, and that node then showed an error. So I'm baffled now.
TigerFox57
2024-08-13 20:11:36 +0000 UTCI'm very confused. I didn't mess with the workflow until after I had processed an image that was in portrait and it came out in landscape. So I looked at the maths nodes stuff, and swapped them round, and now my images seem to be coming out with the correct aspect ratios. I assume that the numbers in the nodes called "Math Expression" that are connected to the node called "Get Image Size" should be the results of the calculation in the box. However, my input image has dimensions 640 Width and 1173 Height, at least according to Windows file manager, but in the Maths boxes, the one connected to the width output says 1376, and the one connected to the height output says 752. It's bizarre! Am I going mad?
TigerFox57
2024-08-13 19:13:34 +0000 UTCNot seeing that at all here. Do you have that conflicting node installed (YOLO)?
Nerdy Rodent
2024-08-13 18:41:38 +0000 UTCStrange. I'm not seeing that myself?
Nerdy Rodent
2024-08-13 16:52:26 +0000 UTCIf neither of the 3 options for Florence are working, check the output when ComfyUI is starting up, as there could be errors shown there. The GitHub issues tab is also a great place for troubleshooting - https://github.com/kijai/ComfyUI-Florence2/issues?q=sort%3Aupdated-desc+is%3Aissue+is%3Aopen - I've also heard there can be strange issues when using the portable download, so trying on a standard install could be another option!
Nerdy Rodent
2024-08-13 16:35:25 +0000 UTCIf neither of the 3 options for Florence are working, check the output when ComfyUI is starting it as there could be errors shown there.
Nerdy Rodent
2024-08-13 16:34:14 +0000 UTCThanks - I'll check that now!
Nerdy Rodent
2024-08-13 16:32:26 +0000 UTCDearest Nerdy. In the workflow for the florence 2 stuff, the width and height ouputs of the get image size node are connected to the wrong inputs of the maths expressions, so a portrait image comes out landscape, and vice versa.
TigerFox57
2024-08-13 09:39:59 +0000 UTCI give up
ALISON REIS
2024-08-13 00:13:21 +0000 UTCI did all of this and I've been at it for 2 days trying to resolve it. It installs but still gives an error: Error occurred when executing DownloadAndLoadFlorence2Model: No package metadata was found for flash_attn File "C:\ComfyUI_windows_portable\ComfyUI\execution.py", line 152, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ I'm very frustrated, I signed this Patreon just because I really liked the workflow and what was shown, but this is a point that should be addressed in the video considering so many people have this problem.
ALISON REIS
2024-08-12 23:55:23 +0000 UTCTry uninstalling and re-installing again using ComfyUI manager
Nerdy Rodent
2024-08-12 18:37:31 +0000 UTCCan't seem to get the ays+ scheduler. Tried messing with other KSamplers and some gave AYS options but not a single 'Latent' node output and not the ays+ scheduler. Is the scheduler super important to the workflow? EDIT: figured id try git clone again since the ppm folder seemed awfully empty and sure enough i didnt clone in admin cmd first try. But this still didn't resolve the issue. Failed to validate prompt for output 9: * KSampler 145: - Value not in list: scheduler: 'ays+' not in ['normal', 'karras', 'exponential', 'sgm_uniform', 'simple', 'ddim_uniform', 'beta']
Brendan Commons
2024-08-12 10:13:24 +0000 UTCtry a different attention option in the Florence node, like eager or sdpa, worked for me when i first started using Florence
Kryptonic
2024-08-12 00:41:18 +0000 UTCyou can also change the attention option in the node to one of the other options instead of flash attention 2, still seems to work fine. Had that problem awhile back when I first tried out Florence.
Kryptonic
2024-08-12 00:02:14 +0000 UTCsame here
ALISON REIS
2024-08-11 18:27:18 +0000 UTCIn short, I compiled Flash Attention 2 from source: pip install flash-attn --no-build-isolation I'm using Cuda 12.4 and pytorch 2.4.0+cu124 For compilation on Windows 11, you need Visual Studio (I used 2022), ninja, packaging. Compilation took more then an hour. I used ChatGPT along the way and that was super helpful: https://chatgpt.com/share/a1c074d9-49af-439f-a0fb-201e5185fec6 (it also contains some unrelated questions, I will remove the chat log in the future)
Remco Bos
2024-08-11 18:11:27 +0000 UTCWhich version of CUDA be you running my friend? I have managed to install it but its not being recognised as installed. Do you have to have a certain version of pytorch & CUDA?
Mark
2024-08-11 17:47:36 +0000 UTCSame issue here. Been trying to install it but it can't build the wheel. Tried installing it using comfyui manager's "install pip packages," which seemed more promising, but still failed.
J
2024-08-11 17:44:11 +0000 UTCCan you tell me how you did it? I get this error message from Comfy: Error occurred when executing DownloadAndLoadFlorence2Model: FlashAttention2 has been toggled on, but it cannot be used due to the following error: the package flash_attn seems to be not installed. Please refer to the documentation of https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2 to install Flash Attention 2. Thank you in Advacne
Jacob Juul
2024-08-10 21:11:04 +0000 UTCFinally got it to work after fighting cuda, flash-attention-2 compilation (windows). it's awesome!
Remco Bos
2024-08-10 20:11:37 +0000 UTCNever mind. I ran out of disk space and reinstalled ComfyUI on another disk and all is well.
Nibmeister
2024-08-10 19:23:23 +0000 UTCFound Get Image Size (duh! in your notes) but not CR Latent.
Nibmeister
2024-08-10 17:00:14 +0000 UTCI can't find the nodes CR Latent and Get Image Size. Where to get?
Nibmeister
2024-08-10 16:43:29 +0000 UTCFor any node you don't need, left-click on it and then press the "delete" key to remove it from the workflow. Or, to remove an area of nodes all at once, hold down "ctrl" along with the left mouse button to create a selection box! Remove the IMG2IMG section and you've basically got the same as setting the switches to 1,2,1/3. I try to organise things into groups too, so another option is to simply use bypass.
Nerdy Rodent
2024-08-10 12:43:45 +0000 UTCIs there a way to make a workflow with only Flux and the LLM party? I don't have a need for the img2img part and im also having the flash attention 2 error. I get errors when updating "pip install -U flash-attn --no-build-isolation".
Lorenzo Gielissen
2024-08-10 12:13:25 +0000 UTCI was getting the very same black image when using sdpa. I installed Flash Attention using the command "pip install flash-attn" and now everything works like a charm.
Lúcio Nunes
2024-08-09 14:29:32 +0000 UTCNot tried it in Python 3.10. I'm using 3.11 here. Microsoft Windows will also be an issue. For issues specific to flash attention, you can check out https://github.com/Dao-AILab/flash-attention
Nerdy Rodent
2024-08-09 13:46:40 +0000 UTCi tried with sdpa, i just got black image :(
p tjie
2024-08-09 11:23:50 +0000 UTCif i install flash attention 2, i got this error: C:\StabilityMatrix\Packages\ComfyUI\venv\Scripts>pip install -U flash-attn --no-build-isolation Collecting flash-attn Downloading flash_attn-2.6.3.tar.gz (2.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 6.1 MB/s eta 0:00:00 Preparing metadata (setup.py) ... error error: subprocess-exited-with-error × python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [12 lines of output] Traceback (most recent call last): File "", line 2, in File "", line 14, in File "C:\StabilityMatrix\Packages\ComfyUI\venv\lib\site-packages\setuptools\__init__.py", line 8, in import _distutils_hack.override # noqa: F401 File "C:\StabilityMatrix\Packages\ComfyUI\venv\lib\site-packages\_distutils_hack\override.py", line 1, in __import__('_distutils_hack').do_override() File "C:\StabilityMatrix\Packages\ComfyUI\venv\lib\site-packages\_distutils_hack\__init__.py", line 70, in do_override ensure_local_distutils() File "C:\StabilityMatrix\Packages\ComfyUI\venv\lib\site-packages\_distutils_hack\__init__.py", line 57, in ensure_local_distutils assert '_distutils' in core.__file__, core.__file__ AssertionError: C:\StabilityMatrix\Packages\ComfyUI\venv\Scripts\python310.zip\distutils\core.pyc [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. any Idea how to solve it? Thank you in Advacne
p tjie
2024-08-09 11:20:44 +0000 UTCThank you!
Daniel Aleshinloye
2024-08-09 11:04:44 +0000 UTCComfyUI-ppm is the one you want for ays+ without using the custom sampler. See the notes in the workflow for more info!
Nerdy Rodent
2024-08-09 07:16:32 +0000 UTCNice!
Nerdy Rodent
2024-08-09 07:15:26 +0000 UTCHow do we add the ays+ scheduler to the ksampler?
Daniel Aleshinloye
2024-08-08 23:34:59 +0000 UTCThe same here, the installation is stuck with this command. I found this https://github.com/kijai/ComfyUI-Florence2/issues/2 Look like they just quit to use it. I have no idea to fix it.
Tapiocapioca
2024-08-08 20:38:13 +0000 UTCFor anyone with limited vram and don't want to run a llm locally give openai a try with their new GPT-4o-mini model. It's $0.15 per million input tokens and $0.60 per million output tokens. Around 300 requests, using it in comfyui, over the last day or so has cost me 3 cents.
troyau
2024-08-08 19:53:39 +0000 UTCYou'll need to do both of these steps before you can run the workflow: 1. Update ComfyUI - E.g. ComfyUI: 2489[8115d8](2024-08-07), Manager: V2.48.5 2. Download the checkpoint - https://huggingface.co/Comfy-Org/flux1-dev/blob/main/flux1-dev-fp8.safetensors - into your ComfyUI models/checkpoints directory. You can make sure you've got the right file via the SHA256 and model size - SHA256: 8e91b68084b53a7fc44ed2a3756d821e355ac1a7b6fe29be760c1db532f3d88a Size of remote file: 17.2 GB. On windows, that would be something like "certutil -hashfile C:\file\path\to\flux1-dev-fp8.safetensors SHA256"
Nerdy Rodent
2024-08-08 19:08:11 +0000 UTCYup, "pip install -U flash-attn --no-build-isolation" should get you the latest version if you need it!
Nerdy Rodent
2024-08-08 19:02:50 +0000 UTCYup, all the options seem pretty much the same to me, so if one doesn't work on your setup then pick another! "pip install flash-attn --no-build-isolation" should get the latest version if you need it - https://github.com/Dao-AILab/flash-attention
Nerdy Rodent
2024-08-08 19:01:40 +0000 UTC"P:\AI\ComfyUI\nodes.py:1446: RuntimeWarning: invalid value encountered in cast img = Image.fromarray(np.clip(i, 0, 255).astype(np.uint8)) Prompt executed in 1681.19 seconds" Well that was a fun 30 minutes.
What to watch high
2024-08-08 16:33:55 +0000 UTC"Error occurred when executing CheckpointLoaderSimple: ERROR: Could not detect model type of: P:\AI\ComfyUI\models\checkpoints\FLUX1\flux1-dev-fp8.safetensors" Comfy is up to date. Any ideas?
What to watch high
2024-08-08 16:00:54 +0000 UTCHi NerdyR I got the same error regarding flash Attention 2 and have been trying for hours to install it to no avail, can anyone help with that. however I did switch to sdpa and it worked then. FlashAttention2 has been toggled on, but it cannot be used due to the following error: the package flash_attn seems to be not installed. Please refer to the documentation of https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2 to install Flash Attention 2.
Stephen Purvis
2024-08-08 14:24:25 +0000 UTCHad the same error. Just change the attention to sdpa, seems to work fine.
troyau
2024-08-08 12:45:13 +0000 UTCi got ollama installed and started, but still get error regarding to "flash attention 2" would someone help me to install "flash attention 2" please?
p tjie
2024-08-08 11:58:31 +0000 UTCi got flashattention2 error in the second json-file ... would you show me how to insatll it please? Error occurred when executing DownloadAndLoadFlorence2Model: FlashAttention2 has been toggled on, but it cannot be used due to the following error: the package flash_attn seems to be not installed. Please refer to the documentation of https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2 to install Flash Attention 2.
p tjie
2024-08-08 09:19:26 +0000 UTCYou are a scholar and a gentleman. Thank you.
Asyouare
2024-08-08 09:00:00 +0000 UTCIt’s the first json in the list. Patreon does appear to take a second to get use to where the links are. I had the same issue first day here too.
Oliver Baddeley
2024-08-08 08:57:31 +0000 UTCAny chance you gonna open up a Test Rodents feed? Maybe a day or two ahead of your YouTube release so that we can help you iron out the little issues like these before the whole world sees your glorious rodentness?
Oliver Baddeley
2024-08-08 08:54:11 +0000 UTCSorry to bother, ss the "LLm party" workflow not included?
Asyouare
2024-08-08 08:44:39 +0000 UTCYup, that's the one :)
Nerdy Rodent
2024-08-08 07:46:00 +0000 UTCi found it, install ComfyUI-ppm with the manager
p tjie
2024-08-08 07:43:59 +0000 UTCThe ComfyUI PPM extra stuff was covered a few weeks back now! More info at https://www.patreon.com/posts/controlnet-union-108501230
Nerdy Rodent
2024-08-08 07:43:45 +0000 UTCmissing it too :(
p tjie
2024-08-08 07:33:42 +0000 UTCsame thing here... missing the ays+ scheduler thing.
Lasse Koivisto
2024-08-08 07:09:08 +0000 UTCGot it working, but you are using the ays sampler... I'm not sure how to install that... I'm Googling, but thought I'd ask like a real pesky rat.
Oliver Baddeley
2024-08-07 22:21:16 +0000 UTC
