Windows¶
Kill GPU process in Windows:
Delete folder
rm -Force .\downloads\
Windows WSL2 Linux Setup Tutorial¶
Install WSL2¶
Ref WSL2 install
wsl --version
wsl --update
wsl --list
wsl.exe --list --online
Select one distribution, and install
C:\Users\lkk>wsl.exe --install Ubuntu-22.04
Note
- If you see this error: “WslRegisterDistribution failed with error: 0x80370114 Error: 0x80370114 The operation could not be started because a required feature is not installed. “ This error message indicates that a required feature is not installed on your system. You can try the following steps to resolve this issue:
Open Control Panel -> Programs and Features -> Turn Windows Feature on or off -> Check Virtual Machine Platform and Windows Subsystem for Linux.
Restart your computer.
Turn on these Windows features:
When your wsl2 installation is finished, you can use this command to enter into the wsl linux:
wsl #enter default wsl distribution
wsl --distribution Ubuntu-22.04 --user lkk #enter a specific distribution
You can access the WSL2 Linux file from the Windows Explorer by accessing the following address “\wsl$”
You can check the version of WSL and perform conversion from WSL2 to WSL1 if needed
C:\Users\lkk>wsl --list --verbose
NAME STATE VERSION
* Ubuntu-22.04 Running 2
Ubuntu-20.04 Running 2
C:\Users\lkk>wsl --set-version Ubuntu-20.04 1
Conversion in progress, this may take a few minutes.
The operation completed successfully.
C:\Users\lkk>wsl --setdefault Ubuntu20.04
The operation completed successfully.
Use WSL for Development¶
Start the WSL Linux via command:
wsl -d distroname -u username
Access files inside the WSL from Windows side: access “\wsl$” in the address bar of Windows File Explorer.
Open WSL in VScode for development: in “Remote Explorer” tab, select “WSL” as the remote the target, connect the target.
How to SSH into WSL2 on Windows from an external machine: https://www.hanselman.com/blog/how-to-ssh-into-wsl2-on-windows-10-from-an-external-machine
Export and Import WSL to a different folder¶
If you want to move the WSL to a different folder or drive to save the space in drive C, you can export the WSL distribution to a local file first
H:\WSL>wsl --export Ubuntu-20.04 ./Ubuntu2004_export.tar
Export in progress, this may take a few minutes.
The operation completed successfully.
After the export process is finished, you can unregister the distribution (the used disk space will become available)
H:\WSL>wsl --unregister Ubuntu-20.04
You can import the distribution again in a new driver (e.g., in H: drive)
H:\WSL>wsl --import Ubuntu20.04 ./Ubuntu20 ./Ubuntu2004_export.tar
Import in progress, this may take a few minutes.
The operation completed successfully.
H:\WSL> wsl --setdefault Ubuntu20.04
H:\WSL>wsl -u lkk #use user "lkk" to login, wsl -d distroname -u username
You can also export the docker data if needed
(base) PS D:\WSL> wsl --list --verbose
NAME STATE VERSION
* Ubuntu22.04 Running 2
docker-desktop Running 2
docker-desktop-data Running 2
(base) PS D:\WSL> wsl --export docker-desktop-data ./docker-desktop-data.tar
The operation completed successfully.
(base) PS D:\WSL> wsl --unregister docker-desktop-data
The operation completed successfully.
(base) PS D:\WSL> wsl --import docker-desktop-data ./docker ./docker-desktop-data.tar
After the export and import a WSL2 distro manually, running that distro on the new drive will end up logging you in as root. You can manually setup the login user via “wsl -u USERNAME” to select a specific user. You can also setup the wsl.conf file to setup a default login user. Run your distro, and then edit /etc/wsl.conf and add a [user] section like this:
[user]
default=lkk
This is the ideal way to set your WSL distro’s default user for imported tars because it’s stored inside the Linux file system and the setting will stick around when you export/import later on.
Install CUDA inside WSL Linux¶
Verify CUDA-capable GPU, check environment, and install gcc
lspci | grep -i nvidia # Did not show anything in WSL Linux
uname -m && cat /etc/*release
gcc --version # install gcc if not available
sudo apt-get update
sudo apt install gcc
uname -r #check kernel version
5.15.90.1-microsoft-standard-WSL2
Download CUDA11.8 Linux WSL2 (the download path is the same to Ubuntu22.04 cuda11.8):
wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run
sudo sh cuda_11.8.0_520.61.05_linux.run
- After CUDA installation, please make sure that
PATH includes /usr/local/cuda-xx.x/bin
LD_LIBRARY_PATH includes /usr/local/cuda-xx.x/lib64
Add the following code to “~/.bashrc”
lkk@Alienware-LKKi7G8:~$ nano ~/.bashrc
export LD_LIBRARY_PATH=/usr/local/cuda-12.1/lib64:$LD_LIBRARY_PATH
export PATH=/usr/local/cuda-12.1/bin:$PATH
lkk@Alienware-LKKi7G8:~$ source ~/.bashrc
#install other required softwares in WSL Linux
$ sudo apt install python3-pip
$ sudo apt install net-tools
$ sudo apt install openssh-server
$ sudo apt install x11-apps
$ curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o Miniconda3-latest-Linux-x86_64.sh
$ bash Miniconda3-latest-Linux-x86_64.sh
$ sudo apt-get update && sudo apt-get install libgl1
Other Python Packages¶
conda install -c conda-forge configargparse
pip install -U albumentations
pip install spconv-cu118 #check installation via import spconv
pip install SharedArray
pip install numba
$ pip install requests
$ pip install tensorboardX
pip install tensorboardX
$ pip install easydict
$ pip install gpustat
$ pip install --upgrade autopep8
$ pip install pyyaml scikit-image onnx onnx-simplifier
$ pip install onnxruntime
$ pip install onnx_graphsurgeon --index-url https://pypi.ngc.nvidia.com
(mypy310) lkk@Alienware-LKKi7G8:~/Developer/3DDepth$ python3 setup.py develop
The detected CUDA version (12.1) mismatches the version that was used to compile
PyTorch (11.8). Please make sure to use the same CUDA versions.
To solve the CUDA mismatch problem, uncomment the CUDA12.1 installation in “~/.bashrc”, install CUDA11.8 in Conda
(mypy310) lkk@Alienware-LKKi7G8:~/Developer/3DDepth$ conda install -c conda-forge cudatoolkit=11.8.0
(mypy310) lkk@Alienware-LKKi7G8:~/Developer/3DDepth$ conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
(mypy310) lkk@Alienware-LKKi7G8:~/Developer/3DDepth$ python3 setup.py develop
Using /home/lkk/miniconda3/envs/mypy310/lib/python3.10/site-packages
Finished processing dependencies for mydetector3d==0.1
(mypy310) lkk@Alienware-LKKi7G8:~/Developer/3DDepth$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
Install huggingface¶
https://huggingface.co/docs/transformers/installation https://huggingface.co/docs/accelerate/basic_tutorials/install
% conda install -c conda-forge accelerate
% accelerate config
In which compute environment are you running?
This machine
------------------------------------------------------------------------------------------------------------------------Which type of machine are you using?
No distributed training
Do you want to run your training on CPU only (even if a GPU / Apple Silicon device is available)? [yes/NO]:NO
Do you wish to optimize your script with torch dynamo?[yes/NO]:NO
Do you want to use DeepSpeed? [yes/NO]: NO
What GPU(s) (by id) should be used for training on this machine as a comma-seperated list? [all]:all
------------------------------------------------------------------------------------------------------------------------Do you wish to use FP16 or BF16 (mixed precision)?
bf16
accelerate configuration saved at /home/lkk/.cache/huggingface/accelerate/default_config.yaml
% accelerate env
% conda install -c huggingface transformers
% conda install -c huggingface -c conda-forge datasets #pip install datasets
% pip install evaluate
% pip install cchardet
% conda install -c conda-forge umap-learn #pip install umap-learn
% conda install sentencepiece
% pip install portalocker
% pip install huggingface_hub
% pip install torchdata
% pip install torchtext
% pip install git+https://github.com/huggingface/transformers
Successfully uninstalled transformers-4.31.0
Successfully installed transformers-4.33.0.dev0
% pip uninstall tokenizers
% pip install tokenizers -U
pip install tensorboard
pip install xformers #installed torch==2.0.1, nvidia-cuda-cupti-cu11==11.7.101, nvidia-nccl-cu11==2.14.3, nvidia-cusparse-cu11==11.7.4.91, nvidia-cuda-runtime-cu11==11.7.99
pip install --upgrade click
pip install numpy==1.23.5
pip install --upgrade tqdm
pip install --upgrade requests
pip install --upgrade jinja2
pip install moviepy
pip install -U "transformers==4.38.1" --upgrade
pip install sentencepiece
pip install timm
pip install albumentations
Install MMdetection3D:
pip install -U openmim
mim install mmengine
mim install 'mmcv>=2.0.0rc4'
mim install 'mmdet>=3.0.0'
(mycondapy310) PS D:\Developer> git clone https://github.com/open-mmlab/mmdetection3d.git
cd mmdetection3d/
pip install -v -e .
pip install cumm-cu118
pip install spconv-cu118
error: Microsoft Visual C++ 14.0 or greater is required. Get it with “Microsoft C++ Build Tools”: https://visualstudio.microsoft.com/visual-cpp-build-tools/
Install langchain
(mypy310) lkk@Alienware-LKKi7G8:~/Developer$ conda install langchain -c conda-forge
(mypy310) lkk@Alienware-LKKi7G8:~/Developer$ pip install langchain[llms]
(mypy310) lkk@Alienware-LKKi7G8:~/Developer$ pip install openai
pip install pypdf
pip install tiktoken
conda install -c conda-forge faiss
pip install unstructured
pip install pdf2image
pip install pdfminer
pip install pdfminer.six
Windows CMD command line:
tar -xf archive.zip #unzip a file