Error:client error: nfc storage connection is unavailable. failed to create nfc downstream
After installing Veeam at a clients site and setting it up to replicate a 2 host cluster and san to a single host with storage it tested OK in the test environment. The servers where then IP`d and put into the correct sub net with the relevant gateways etc.
Then the replication job was made live and left to run over night. Next morning I checked only to find this error below.
error:client error: NFC Storage connection is unavailable. Failed To Create NFC Downstream
Now I knew that IP`s had been changed so I tested the following.
From the Veeam server.
Pinged its gateway – OK.
Pinged its DNS servers – OK.
Pinged the source VM hosts – FAIL
Pinged the storage – OK.
The the Veeam side failed on pinging the source hosts, I checked all my local settings which were ok. I went onto my hosts and checked the Local IP settings. It turns out that there was incorrect setting for the hosts gateway. I changed accordingly and the pings were fine.The hosts could not talk back to the Veeam server correctley. Then I set the replication away and it completed successfully.
The point of this post is the error:client error: NFC Storage connection is unavailable. Failed To Create NFC Downstream is generally an IP misconfiguration error or a communications error. So look there first and test the simple tings like pings.
Tags: Storage
Allen is an IT Consultant and holds the following accreditations. MCSA, MCSE, MCTS, MCITP, CCA, CCSP, VCP 4,5, 6 and HP ASE, AIS — Network Infrastructure.
Veeam error nfc storage connection | .ua
Solution
Caution: These steps expose the security vulnerabilities with SSLv3. This issue is resolved in VMware View 6.2, available at VMware Downloads. For more information, see VMware Horizon 6 version 6.2 Release Notes.
The SSLv3 support can be enabled for these ports and services:
CIM Port 5989
Authd Service Port 902
Enabling support for SSLv3 on CIM Port 5989 in ESXi
Create a backup copy of the /etc/sfcb/sfcb.cfg file.
Edit the /etc/sfcb/sfcb.cfg file to append the following line at the end of the file:
enableSSLv3: true
Note: If you have the line enableSSLv3: false in the file, change it to enableSSLv3: true
For Example:
cat /etc/sfcb/sfcb.cfg
# Generated by sfcb-config.py. Do not modify this header.
# VMware ESXi 6.0.0 build-3029758
#
basicAuthLib: sfcBasicPAMAuthentication
certificateAuthLib: sfcCertificateAuthentication
cimXmlFdHardLimit: 1024
cimXmlFdSoftLimit: 512
.
.
.
threadStackSize: 524288
useChunking: true
sslCipherList: HIGH:!DES-CBC3-SHA!CAMELLIA128-SHA!CAMELLIA256-SHA
enableSSLv3: true
Restart the SFCBD service with the command:
/etc/init.d/sfcbd-watchdog restart
Enabling support for SSLv3 on Authd service 902 in ESXi
Create a backup copy of the /etc/vmware/config file
Edit the /etc/vmware/config file to append the following line at the end of the file:
vmauthd.ssl.noSSLv3 = “false”
Note: If you have the line vmauthd.ssl.noSSLv3 = “true” in the file, change it to vmauthd.ssl.noSSLv3 = “false”
For Example:
cat /etc/vmware/config
libdir = “/usr/lib/VMware”
authd.proxy.nfc = “vmware-hostd:ha-nfc”
authd.proxy.nfcssl = “vmware-hostd:ha-nfcssl”
authd.proxy.vpxa-nfcssl = “vmware-vpxa:vpxa-nfcssl”
authd.proxy.vpxa-nfc = “vmware-vpxa:vpxa-nfc”
authd.fullpath = “/sbin/authd”
vmauthd.ssl.noSSLv3 = “false”
Restart the rhttpproxy service with the command:
/etc/init.d/rhttpproxy restart
Additional Information
For the related Veeam Knowledge Base article, see http://www.veeam.com/kb2063.
Follow these steps to enable SSLv3 protocol on hostd service for ESXi 6.0 U1b later.
By default SSLv3 is disabled. If you want to enable SSLv3, set the setting to empty by using the below command:
Login to ESXi through SSH.
Run the following command to get a list of disabled protocols for hostd:
esxcli system settings advanced list -o /UserVars/VMAuthdDisabledProtocols
Path: /UserVars/VMAuthdDisabledProtocols
Type: string
Int Value: 0
Default Int Value: 0
Min Value: 0
Max Value: 0
String Value:sslv3
Default String Value: sslv3
Valid Characters: *
Description: VMAuthd disabled protocols. Choices are sslv3, tlsv1, tlsv1.1, tlsv1.2. By default sslv3 is disabled. If no protocol is specified, all protocols are enabled.
If SSLv3 is disabled, To enable SSLv3 is run the following command:
esxcli system settings advanced set -o /UserVars/VMAuthdDisabledProtocols -s “”
Restart the rhttpproxy services by running the following command:
/etc/init.d/rhttpproxy restart
Run the following command to get a list of enabled protocols for hostd:
esxcli system settings advanced list -o /UserVars/VMAuthdDisabledProtocols
Path: /UserVars/VMAuthdDisabledProtocols
Type: string
Int Value: 0
Default Int Value: 0
Min Value: 0
Max Value: 0
String Value:
Default String Value: sslv3
Valid Characters: *
Description: VMAuthd disabled protocols. Choices are sslv3, tlsv1, tlsv1.1, tlsv1.2. By default sslv3 is disabled. If no protocol is specified, all protocols are enabled.
[collapse]
Veeam: nfc storage connection is unavailable.failed to create nfc download stream. – zewwy’s info tech talks
I’ll keep this post short as well.
Run replication Job… ERROR. Check error, huh haven’t this one before…
4/16/2021 11:22:16 AM :: Processing VMName Error: NFC storage connection is unavailable. Storage: [stg:datastore-3,nfchost:host-2,conn:vcenter]. Storage display name: [ESXi-Local-Datastore]. Failed to create NFC upload stream. NFC path: [nfc://conn:vCenter,nfchost:host-23982,stg:datastore-23983@VMName_replica/VMName.vmx].
To note on this, I did some changes, I changed a route between sites (as I needed to reduce a entire network from being improperly routed, but some of services still required access from the main location, thus some dedicated /32 routes were put in their place).
I had also just patched the host in question and was testing jobs after the patch. Since I wasn’t exactly sure which was the cause. I decided to do regular troubleshooting to get more details to the root cause.
I love Veeam, they got a nice KB to help with this. So I followed along, checking the main Veeam server log areas didn’t have the log file in question, so was pretty confident it was still using the proxy at the alt site.
Checking the log as mentioned by the KB, sure enough the same error line showed up which indicated it was DNS related. Checking the proxie’s DNS settings…. DOH. It was using a server within the routes I had removed, and didn’t create a dedicated /32 for, as I wasn’t expecting any systems here to need to communicate to that subnet.
Now that I know what the issue is… this feels familiar… oh yeah the Veeam Soap Fault issue I had to deal with…
The funny part about this is… 1) it’s the same server/proxy 2) Again DNS related 3) Again going to stick with host file to avoid dependencies on DNS servers
In my case the error showed the ESXi server by the hostname WAS fully qualified, but access to a DNS server to resolve it was unavailable. As soon as I saw this I had two options:
- Create a route to allow the Proxy to reach required DNS servers (which won’t be available in a DR case) OR
- Just add a static record in the Proxy host file. (DNS server not required, but if hostname or IP changes needs to be adjusted manually here)
As you can see I have the exactly 2 options similar to my first post, the difference is now it was fully dependent on DNS. Since this is a self hosted instance of a Veeam proxy, there’s a good chance DNS access might not be available when time comes, so this option was chosen.
It’s important to note that when these types of choices are made it is well documented WHY they were made.
So in this case… to resolve it I added a record in the Proxy’s local host file
172.x.x.x ESXi.domain.postfix
You may notice that the ESXi hostname is not within the initial error, it only tells your the datastore, the Veeam logs will indicate which lookup failed. More than likely the hostname look up for the ESXi host in which the VM will be created on.
I really hope this post helps someone. Honestly I just followed the Veeam KB which was a great source reference to troubleshooting the issue. Your case maybe different, depending on the root cause your resolution maybe different then what was discussed here.
Cheers stay safe everyone.
§
I’ll keep this post short as well.
Run replication Job… ERROR. Check error, huh haven’t this one before…
4/16/2021 11:22:16 AM :: Processing VMName Error: NFC storage connection is unavailable. Storage: [stg:datastore-3,nfchost:host-2,conn:vcenter]. Storage display name: [ESXi-Local-Datastore]. Failed to create NFC upload stream. NFC path: [nfc://conn:vCenter,nfchost:host-23982,stg:datastore-23983@VMName_replica/VMName.vmx].
To note on this, I did some changes, I changed a route between sites (as I needed to reduce a entire network from being improperly routed, but some of services still required access from the main location, thus some dedicated /32 routes were put in their place).
I had also just patched the host in question and was testing jobs after the patch. Since I wasn’t exactly sure which was the cause. I decided to do regular troubleshooting to get more details to the root cause.
I love Veeam, they got a nice KB to help with this. So I followed along, checking the main Veeam server log areas didn’t have the log file in question, so was pretty confident it was still using the proxy at the alt site.
Checking the log as mentioned by the KB, sure enough the same error line showed up which indicated it was DNS related. Checking the proxie’s DNS settings…. DOH. It was using a server within the routes I had removed, and didn’t create a dedicated /32 for, as I wasn’t expecting any systems here to need to communicate to that subnet.
Now that I know what the issue is… this feels familiar… oh yeah the Veeam Soap Fault issue I had to deal with…
The funny part about this is… 1) it’s the same server/proxy 2) Again DNS related 3) Again going to stick with host file to avoid dependencies on DNS servers
In my case the error showed the ESXi server by the hostname WAS fully qualified, but access to a DNS server to resolve it was unavailable. As soon as I saw this I had two options:
- Create a route to allow the Proxy to reach required DNS servers (which won’t be available in a DR case) OR
- Just add a static record in the Proxy host file. (DNS server not required, but if hostname or IP changes needs to be adjusted manually here)
As you can see I have the exactly 2 options similar to my first post, the difference is now it was fully dependent on DNS. Since this is a self hosted instance of a Veeam proxy, there’s a good chance DNS access might not be available when time comes, so this option was chosen.
It’s important to note that when these types of choices are made it is well documented WHY they were made.
So in this case… to resolve it I added a record in the Proxy’s local host file
172.x.x.x ESXi.domain.postfix
You may notice that the ESXi hostname is not within the initial error, it only tells your the datastore, the Veeam logs will indicate which lookup failed. More than likely the hostname look up for the ESXi host in which the VM will be created on.
I really hope this post helps someone. Honestly I just followed the Veeam KB which was a great source reference to troubleshooting the issue. Your case maybe different, depending on the root cause your resolution maybe different then what was discussed here.
Cheers stay safe everyone.
