我编写了以下代码，以XML响应的形式获取结果，并将一些内容写入Java的文件中。这是通过向公共数据库接收大约700,000个查询的XML响应来完成的。
但是，在代码可以写入文件之前，它会被代码中随机位置的一些随机异常（来自服务器）停止。我尝试从For-loop本身写入文件，但是无法做到。所以我尝试将收到的响应中的块存储到Java HashMap中，并在一次调用中将HashMap写入文件。但是在代码收到for循环中的所有响应并将它们存储到HashMap之前，它会停止一些异常（可能在第15000次迭代!!）。当需要这样的迭代来获取数据时，是否还有其他有效的方法可以用Java写入文件？
我用于此代码的本地文件是here。
我的代码是，
import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.io.Reader;
import java.io.StringWriter;
import java.net.URL;
import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.transform.OutputKeys;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
import org.json.XML;
import org.w3c.dom.Document;
import org.xml.sax.SAXException;
public class random {
static FileWriter fileWriter;
static PrintWriter writer;
public static void main(String[] args) {
// Hashmap to store the MeSH values for each PMID
Map<String, String> universalMeSHMap = new HashMap<String, String>();
try {
// FileWriter for MeSH terms
fileWriter = new FileWriter("/home/user/eclipse-workspace/pmidtomeshConverter/src/main/resources/outputFiles/pmidMESH.txt", true);
writer = new PrintWriter(fileWriter);
// Read the PMIDS from this file
String filePath = "file_attached_to_Post.txt";
String line = null;
BufferedReader bufferedReader = new BufferedReader(new FileReader(filePath));
String[] pmidsAll = null;
int x = 0;
try {
//print first 2 lines or all if file has less than 2 lines
while(((line = bufferedReader.readLine()) != null) && x < 1) {
pmidsAll = line.split(",");
x++;
}
}
finally {
bufferedReader.close();
}
// List of strings containing the PMIDs
List<String> pmidList = Arrays.asList(pmidsAll);
// Iterate through the list of PMIDs to fetch the XML files from PubMed using eUtilities API service from PubMed
for (int i = 0; i < pmidList.size(); i++) {
String baseURL = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pubmed&retmode=xml&rettype=abstract&id=";
// Process to get the PMIDs
String indPMID_p0 = pmidList.get(i).toString().replace("[", "");
String indPMID_p1 = indPMID_p0.replace("]", "");
String indPMID_p2 = indPMID_p1.replace("\\", "");
String indPMID_p3 = indPMID_p2.replace("\"", "");
// Fetch XML response from the eUtilities into a document object
Document doc = parseXML(new URL(baseURL + indPMID_p3));
// Convert the retrieved XMl into a Java String
String xmlString = xml2String(doc); // Converts xml from doc into a string
// Convert the Java String into a JSON Object
JSONObject jsonWithMeSH = XML.toJSONObject(xmlString); // Converts the xml-string into JSON
// -------------------------------------------------------------------
// Getting the MeSH terms from a JSON Object
// -------------------------------------------------------------------
JSONObject ind_MeSH = jsonWithMeSH.getJSONObject("PubmedArticleSet").getJSONObject("PubmedArticle").getJSONObject("MedlineCitation");
// List to store multiple MeSH types
List<String> list_MeSH = new ArrayList<String>();
if (ind_MeSH.has("MeshHeadingList")) {
for (int j = 0; j < ind_MeSH.getJSONObject("MeshHeadingList").getJSONArray("MeshHeading").length(); j++) {
list_MeSH.add(ind_MeSH.getJSONObject("MeshHeadingList").getJSONArray("MeshHeading").getJSONObject(j).getJSONObject("DescriptorName").get("content").toString());
}
} else {
list_MeSH.add("null");
}
universalMeSHMap.put(indPMID_p3, String.join("\t", list_MeSH));
writer.write(indPMID_p3 + ":" + String.join("\t", list_MeSH) + "\n");
System.out.println("Completed iteration for " + i + " PMID");
}
// Write to the file here
for (Map.Entry<String,String> entry : universalMeSHMap.entrySet()) {
writer.append(entry.getKey() + ":" + entry.getValue() + "\n");
}
System.out.print("Completed writing the file");
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ParserConfigurationException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (SAXException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (TransformerException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
writer.flush();
writer_pubtype.flush();
writer.close();
writer_pubtype.close();
}
}
private static String xml2String(Document doc) throws TransformerException {
TransformerFactory transfac = TransformerFactory.newInstance();
Transformer trans = transfac.newTransformer();
trans.setOutputProperty(OutputKeys.METHOD, "xml");
trans.setOutputProperty(OutputKeys.INDENT, "yes");
trans.setOutputProperty("{http://xml.apache.org/xslt}indent-amount", Integer.toString(2));
StringWriter sw = new StringWriter();
StreamResult result = new StreamResult(sw);
DOMSource source = new DOMSource(doc.getDocumentElement());
trans.transform(source, result);
String xmlString = sw.toString();
return xmlString;
}
private static Document parseXML(URL url) throws ParserConfigurationException, SAXException, IOException {
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder db = dbf.newDocumentBuilder();
Document doc = db.parse((url).openStream());
doc.getDocumentElement().normalize();
return doc;
}
private static String readAll(Reader rd) throws IOException {
StringBuilder sb = new StringBuilder();
int cp;
while ((cp = rd.read()) != -1) {
sb.append((char) cp);
}
return sb.toString();
}
public static JSONObject readJsonFromUrl(String url) throws IOException, JSONException {
InputStream is = new URL(url).openStream();
try {
BufferedReader rd = new BufferedReader(new InputStreamReader(is, Charset.forName("UTF-8")));
String jsonText = readAll(rd);
JSONObject json = new JSONObject(jsonText);
return json;
} finally {
is.close();
}
}
}
这是它在异常之前在控制台上打印的内容。
因此，在循环中的任何随机点，我都会得到以下异常。
java.io.FileNotFoundException：https://dtd.nlm.nih.gov/ncbi/pubmed/out/pubmed_190101.dtd at sun.net.www.protocol.http.HttpURLConnection.getInputStream0（HttpURLConnection.java:1890）at sun.net.www.protocol.http.HttpURLConnection.getInputStream（HttpURLConnection.java:1492）at at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream（HttpsURLConnectionImpl.java:263）位于com.sun的com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity（XMLEntityManager.java:647）。 org.apache.xerces.internal.impl.XMLEntityManager.startEntity（XMLEntityManager.java:1304）位于com.sun的com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startDTDEntity（XMLEntityManager.java:1270）。 org.apache.xerces.internal.impl.XMLDTDScannerImpl.setInputSource（XMLDTDScannerImpl.java:264）位于com的com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl $ DTDDriver.dispatch（XMLDocumentScannerImpl.java:1161）。 sun.com.apache.xerces.internal.impl.XMLDocumentScannerImpl $ DTDDriver.next（XMLDocumentScannerImpl.java:1045）at com.sun.org.apache.xerces .internal.impl.XMLDocumentScannerImpl $ PrologDriver.next（XMLDocumentScannerImpl.java:959），位于com.sun.org.apache的com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next（XMLDocumentScannerImpl.java:602）位于com.sun.org.apache的com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse（XML11Configuration.java:842）中的.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument（XMLDocumentFragmentScannerImpl.java:505） .xerces.internal.parsers.XML11Configuration.parse（XML11Configuration.java:771）at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse（XMLParser.java:141）at com.sun.org.apache位于javax.xml.parsers.DocumentBuilder的com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse（DocumentBuilderImpl.java:339）中的.xerces.internal.parsers.DOMParser.parse（DOMParser.java:243） .parse（DocumentBuilder.java:121）at pmidtomeshConverter.Convert2MeSH.parseXML（Convert2MeSH.java:240）at pmidtomeshConverter.Convert2MeSH.main（Convert2MeSH.java:121）
没有必要使用地图;只需直接写入文件。为了更好的性能使用BufferedWriter。
我还会检查服务器端没有速率限制或任何这种性质（你可以从你得到的错误中猜出）。在解析或下载失败时将响应保存在单独的文件中，这样您就能更好地诊断问题。
我还会投入一些时间来实现重启机制，这样你就可以从最后一个失败的位置重启进程，而不是每次都从头开始。它可以像提供跳过计数器一样简单，作为跳过前N个请求的输入。
你应该重新使用DocumentBuilderFactory，这样它每次都不会加载相同的DTD。此外，您可能希望完全禁用DTD验证（除非您只需要有效文档，在这种情况下，最好捕获该异常并将错误的XML转储到单独的文件中以供查看）。
private static DocumentBuilderFactory dbf;
public static void main(String[] args) {
dbf = DocumentBuilderFactory.newInstance();
dbf.setFeature("http://apache.org/xml/features/nonvalidating/load-external-dtd", false);
dbf.setFeature("http://xml.org/sax/features/validation", false);
...
}
private static Document parseXML(URL url) throws ParserConfigurationException, SAXException, IOException {
DocumentBuilder db = dbf.newDocumentBuilder();
Document doc = db.parse((url).openStream());
doc.getDocumentElement().normalize();
return doc;
}
您希望解析器在解析时忽略DTD。
使用此功能：
dbf.setFeature("http://apache.org/xml/features/nonvalidating/load-external-dtd", false);
有关其他功能，请参阅Xerces documentation。